<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nadim Kawwa on Medium]]></title>
        <description><![CDATA[Stories by Nadim Kawwa on Medium]]></description>
        <link>https://medium.com/@nadimkawwa?source=rss-85e7e8816e35------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*WelqVj_5FXlmVOZTLa3stw.jpeg</url>
            <title>Stories by Nadim Kawwa on Medium</title>
            <link>https://medium.com/@nadimkawwa?source=rss-85e7e8816e35------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:33:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nadimkawwa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running Jupyter Notebooks On Docker Containers]]></title>
            <link>https://medium.com/data-science/running-jupyter-notebooks-on-docker-containers-47862f6b96a0?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/47862f6b96a0</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 04:37:21 GMT</pubDate>
            <atom:updated>2022-08-05T04:37:21.935Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Project with SageMaker Studio Lab and Docker</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t4NdMuJGOuMXXM8q" /><figcaption>Photo by <a href="https://unsplash.com/es/@carrier_lost?utm_source=medium&amp;utm_medium=referral">Ian Taylor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The objective of this post is to run a data science workflow on AWS and then ship it using Docker, thus creating an end-to-end machine learning task.</p><p>Furthermore, I will be focusing more on the “how” to dockerize a data science project, more so than “why this project is cool”. That being said, there are many <a href="https://www.microfocus.com/documentation/enterprise-developer/ed40pu5/ETS-help/GUID-F5BDACC7-6F0E-4EBB-9F62-E0046D8CCF1B.html">benefits for using Docker</a>:</p><ul><li>Portability</li><li>Performance</li><li>Agility</li><li>Isolation</li><li>Scalability</li></ul><p>On the other hand, AWS <a href="https://studiolab.sagemaker.aws/">SageMaker Studio Lab</a> provides the power of SageMaker without the need to explicitly define every subprocess.</p><h3>Picking A Dataset</h3><p>For this project we will be using 34,686,770 Amazon reviews from 6,643,669 users on 2,441,053 products, from the Stanford Network Analysis Project (<a href="https://snap.stanford.edu/data/web-Amazon.html">SNAP</a>). The data distributed under the BSD license and is free for both academic and commercial use.</p><p>The data used is a subset of that data, found on Kaggle. It contains 1,800,000 training samples, 200,000 testing samples, such that each review is either “positive” or “negative”.</p><p><a href="https://www.kaggle.com/datasets/kritanjalijain/amazon-reviews">Amazon reviews</a></p><p>Since this is a large dataset and we are working with AWS, the data is stored in an S3 bucket. In addition, make sure that the bucket and AWS services you’e using are in the same <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html">region</a>. For this project, the region of choice is <strong>us-east-2</strong> (US East Ohio).</p><h3>IAM: Know Your Role</h3><p>Working with AWS, one of the common pitfalls is not having the required permissions, or not knowing the extent of your permissions. Go to <a href="https://aws.amazon.com/iam/">Identity and Access Management </a>(IAM), create a role, and attach the following policies to it:</p><ul><li>AmazonS3FullAcess</li><li>AmazonSageMakerFullAccess</li><li>AWSGlueConsoleSageMakerNotebookFullAccess</li><li>AWSImageBuilderFullAccess</li><li>AmazonSageMakerPipelinesIntegrations</li><li>AmazonSageMakerGroundTruthExecution</li></ul><h3>Why SageMaker Studio Lab?</h3><p>SageMaker Studio Lab provides a lightweight environment to perform tasks faster than you would on SageMaker. <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lab.html">It comes with some drawbacks</a> such as not having access to AWS Data Wrangler and no large scale distributed training.</p><p>To get started with SageMaker Studio Lab, all you need is an AWS account and wait about 24hrs to get your account approved. Once inside the lab, the environment looks like a Jupyter notebook.</p><p>You can request a free account here: <a href="https://studiolab.sagemaker.aws/">https://studiolab.sagemaker.aws/</a></p><h3>Starting A Session</h3><p>The code blurb below is a standard way to connect to your AWS resources and defines essential resources. Think of it a sort of <a href="https://www.in-ulm.de/~mascheck/various/shebang/">Shebang</a> to get started with AWS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04e23055a24ade903742fce9506c9fb5/href">https://medium.com/media/04e23055a24ade903742fce9506c9fb5/href</a></iframe><h3>Preprocessing Steps</h3><p>Inside the SageMaker Studio, the first step is to write a preprocessing script such as the one shown below. This script and all subsequent scripts need to contain all required libraries and be able to run independently.</p><p>Moreover, the mindset to adopt in a ML workflow is “t<em>his will be running on someone else’s machine</em>”. Hence it’s crucial that what is written in a Jupyter Notebook environment can be contained in a concise python script.</p><p>For those getting started with SageMaker, remember to stick to the prescribed directory format: “<strong><em>/opt/ml/…</em></strong>”. In fact, “/opt/ml” and all its subdirectories are reserved by SageMaker <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/build-your-own-processing-container.html">per the documentation</a>. For example, “<a href="https://sagemaker-examples.readthedocs.io/en/latest/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.html#:~:text=%2Fopt%2Fml%2Fmodel%2F,a%20compressed%20tar%20archive%20file.">/opt/ml/model/</a>” is the directory where you write the model that your algorithm generates.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d81b19c5c3f685d5b3acf6763df6f0e/href">https://medium.com/media/1d81b19c5c3f685d5b3acf6763df6f0e/href</a></iframe><h3>Training Script</h3><p>After preprocessing comes the training which like the previous section is contained in a script. The specific model isn’t as important as having the right workflow set up. Once the model is functioning as intended, feel free to plug in your algorithm of choice.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3f43dbe031598d5245fbb821d20ef25/href">https://medium.com/media/e3f43dbe031598d5245fbb821d20ef25/href</a></iframe><h3>Model Evaluation</h3><p>The data is perfectly balanced: 50% positive and 50% negative. Therefore accuracy is a sufficient metric. Precision and recall are also included to uncover any issues in prediction.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f91b9b01fcd4a5e018690a62f879d75/href">https://medium.com/media/9f91b9b01fcd4a5e018690a62f879d75/href</a></iframe><p>Prior to any parameter optimization, the accuracy hovers at 80% and so is precision/recall.</p><h3>Processing Jobs With FrameworkProcessor</h3><p>The SageMaker processing module allows us to add dependencies using a <a href="https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.FrameworkProcessor">FrameWork</a> processor which allows for a set of Python scripts to be run as part of the Processing Job. In addition, it gives the processor access to multiple scripts in a directory instead of only specifying one script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8e36823b89ae74953f38765f97ca7fd/href">https://medium.com/media/d8e36823b89ae74953f38765f97ca7fd/href</a></iframe><h3>Shipping to Docker</h3><p>Now that our workflow is running smoothly, it’s time to ship! First create a directory called docker inside the lab and write the Dockerfile to create the processing container:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a660c4222caf75482024d7c561351fb/href">https://medium.com/media/0a660c4222caf75482024d7c561351fb/href</a></iframe><p>Next step is building the container using the docker command which creates an Elastic Registry Container (ECR) and pushing the docker image:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/666d57b4d8df2644cba4ce21969e4fb0/href">https://medium.com/media/666d57b4d8df2644cba4ce21969e4fb0/href</a></iframe><p><strong>P.S.</strong> If you’re using an Apple M1 computer or newer, make sure to explicitly call out your platform when building the docker image.</p><h3>Running the Docker Container</h3><p>First we call the ScriptProcessor class which lets you run a command inside a container, and now we can run the same scripts from before except they are inside a docker container.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91a420b559549d420c2bffe767730dfe/href">https://medium.com/media/91a420b559549d420c2bffe767730dfe/href</a></iframe><p>And that’s it, we successfully ran a processing job on a docker container. The training and evaluation containers are not shown for brevity since they follow similar steps.</p><h3>Conclusion</h3><p>With AWS services and Docker, we are able to push python code into a container and productize the data science workflow.</p><p>If you’re interested in seeing the whole code along with outputs to validate your own work, check out the Github repo below:</p><p><a href="https://github.com/NadimKawwa/amazon-review-docker/blob/main/README.md">amazon-review-docker/README.md at main · NadimKawwa/amazon-review-docker</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47862f6b96a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/running-jupyter-notebooks-on-docker-containers-47862f6b96a0">Running Jupyter Notebooks On Docker Containers</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design Considerations for A Web App]]></title>
            <link>https://medium.com/data-science/design-considerations-for-a-web-app-4b8ebb58315?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b8ebb58315</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[dota-2]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Sat, 06 Nov 2021 17:39:15 GMT</pubDate>
            <atom:updated>2021-11-06T17:39:15.782Z</atom:updated>
            <content:encoded><![CDATA[<h4>From Jupyter Notebooks to Fully Functional Web App</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYBZoYioCcrs6_7tlcMtQw.jpeg" /><figcaption>Source: <a href="https://dota2.fandom.com/wiki/Custom_Wards">Dota2 Wiki</a>, available under <a href="https://www.fandom.com/licensing">CC BY-NC-SA 3.0</a></figcaption></figure><p>Sometime ago, I created a ward cluster finder for DOTA2 and shared it <a href="https://towardsdatascience.com/where-to-place-wards-in-dota2-84f534b9b64e">here on medium</a> and with the <a href="https://www.reddit.com/r/DotA2/comments/q021rw/i_analyzed_where_pro_players_place_wards/?utm_source=share&amp;utm_medium=web2x&amp;context=3">DOTA2 community on Reddit.</a> The objective was to parse through professional matches and identify warding patterns of the pros. In addition, I wanted to compare my findings versus the suggestions from the paid <a href="https://www.dota2.com/plus">DOTA Plus</a> service.</p><p>I felt accomplished for turning a side project into something of tangible value. In addition to praise, I also received a lot of feedback on how to improve it. The purpose of this post is to explore design considerations for a data science web app, the importance of user feedback, and how to anticipate future demands.</p><p>To skip ahead to the final product, follow the link below:</p><p><a href="https://share.streamlit.io/nadimkawwa/dotawardfinder/main/appWardObjectives.py">https://share.streamlit.io/nadimkawwa/dotawardfinder/main/appWardObjectives.py</a></p><p>For the repository:</p><p><a href="https://github.com/NadimKawwa/DOTAWardFinder">GitHub - NadimKawwa/DOTAWardFinder: Find ward clusters in DOTA</a></p><h3>Who’s The Audience?</h3><p>Before creating anything, it’s important to reflect who is going to use it and how. I expect this project to be mostly used by DOTA2 players, some data scientists, and perhaps data scientists who play DOTA2. This entails that using a lot of technical jargon might be off-putting and should be avoided. Furthermore it helps to put yourself in the audience’s shoes, which in this case means playing some DOTA2 games to gain a good grasp of game mechanics.</p><h3>Get Initial Feedback From Users</h3><p>The benefit of sharing projects on various forums is the feedback from users and suggestions for improvement. It’s perfectly normal to hold off publishing something out of fear that it won’t be perfect, or that there’s a fatal flaw somewhere. Personally, I was afraid that the project was not finished enough to warrant feedback from anyone. However, please keep in mind that we are allowed to fail and critique is a valid way to improve. I ended up deploying the model and spent the next weeks reflecting on the feedback I received.</p><p>There were three ideas that kept reappearing and are implemented in this project. The image below is taken from the original project and shows a sample for warding between minutes 30–40. In the next subsections, I explain the shortcomings of this plot and how I addressed users’ concerns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEAoTgdt6QO3fPJDKfE2gw.png" /><figcaption>Image By Author</figcaption></figure><h4>Less Clusters, More Circles</h4><p>An observer ward provides vision over an area, and a sentry ward reveals invisible units if the user has vision in that area. The earlier plot shows the ward clusters such that each color corresponds to a different cluster (red, orange, pink, etc…).</p><p>I learned that although this showed where the wards were concentrated, it didn’t really show the area that they covered. Users wanted to know: What’s the general area being surveilled? Hence I replaced the clusters with the respective average centroid and drew a circle around it.</p><p>In addition, the color scheme will be the same for all wards that belong to the same team and category. The result is a more condensed representation and less effort for the user to identify warding spots.</p><h4>Warding As A Function of Objectives, Not Time</h4><p>Perhaps the most poignant observation was that not all games follow the same tempo. Indeed, some games are finished in as little as 25 minutes and others take over an hour. The pace of a game is determined by capturing objectives such as towers and barracks.</p><p>Therefore, clustering wards based on time is rather misleading since it does not take into account how far ahead we are in the game. I modified the segmentation process to be in function of objectives rather than time.</p><h4>Less Images Means Less Confusion</h4><p>Some users’ posts said something along the lines of:</p><ul><li>It’s hard to see anything if you’re on a mobile phone.</li><li>The plot titles are small to read.</li><li>Why can’t we see the clusters side by side?</li></ul><p>The solution to this is combine the four small subplots into a bigger, all encompassing plot. It allows users to see everything in a consolidated format and hopefully prevents squinting at small fonts.</p><h4>Implementing the Feedback</h4><p>The result is the plot shown below. An additional enhancement is showing the status of the towers so that the plot is self explanatory. For example, Radiant’s last remaining tower in the top lane is Tier 3 and both teams compete on to have a sentry ward there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmvtyYx9wpNdoqQDJLpSVg.png" /><figcaption>Image by Author</figcaption></figure><h3>Additional Design Considerations</h3><p>User feedback opens the door to reflecting on previous choices. Indeed, once we start changing things we start questioning if entire sections of code need a rewrite. This section goes over these sections of code that were completely rewritten.</p><h4>Computational Effort</h4><p>The clustering algorithm of choice is DBSCAN and can be computationally expensive. In addition, finding the optimal parameters requires several iterations. Last but not least, as the dataset becomes bigger it expounds the computational effort, leading us to question: Should the training algorithm be hosted on a cloud platform?</p><p>The answer is no, not necessarily. Why force the cloud service to compute the same thing over and over again? Moreover, why make the user wait for the computation to finish? The solution is to do all the computations manually and locally once, and upload the static plots on a storage service (AWS S3 comes to mind). This simplifies things since all the code does is fetch an image and show it to the user.</p><h4>Asking Too Much Input From The User</h4><p>The previous version of this project asked the user for three things:</p><ul><li>Match time range</li><li>Epsilon parameter</li><li>Minimum samples parameter</li></ul><p>The last two bullets aren’t very clear to someone with little to no background in data science. Therefore these inputs can cause an unnecessary headache. Indeed, how is someone supposed to guess the optimal parameters without having spent considerable time exploring the data?</p><p>The parameters are replaced by a simple question: Please state what towers are alive at time of warding?</p><h4>Roshan’s Pit</h4><p>Roshan’s pit is a rather peculiar objective, and the game can be won without having to kill Roshan. The latter drops the aegis of the immortal which gives a player reincarnation after death. Moreover, as the game progresses and players grow more powerful, so does Roshan.</p><p>Another interesting fact: Roshan respawns at certain intervals and can be killed multiple times (<a href="https://www.opendota.com/matches/6080190774">as many as 7 times in a Team Liquid game</a>).</p><p>The warding plot for Roshan is a special case and needs to reflect the average state of captured objectives. Therefore, the towers are partially colorized to reflect on average how far along the game has progressed. For example, if 45% of Dire’s tier 2 towers are alive at time of Roshan’s death, then the tower is 45% colored in red and 55% colored in grey.</p><p>Furthermore, wards are typically placed 2 minutes prior to <em>roshing</em> (the act of killing Roshan), and only these wards are considered. An additional simplification is only scanning wards placed in the upper left quadrant of the map.</p><p>The result is the image below, showing a competition for warding the entrances to the pit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D53KpiZV3z9w8c5rb2hMew.png" /><figcaption>Plot by author</figcaption></figure><h4>Edge Cases</h4><p>There are 6 lanes total and each lane can have four states varying from no towers are taken to tier 3 tower is destroyed in that specific lane. This means we have 4096 combinations to consider.</p><p>The previous statement is only partially true since most of these cases are non-existent or have too few data points. Hence it’s helpful to let the user know when they’re dealing with an edge case.</p><h3>Implementation</h3><p>With all the design considerations explained, we can move on to implementation. The simplified solution is a series of sliders that convert user input into a path to the static plot. Simply slide the values to desired state, hit the “Show Wards for Towers” and watch the result manifest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2o4aM539WWOqLpBcRfTAA.png" /><figcaption>Image by Author</figcaption></figure><h3>Bonus: For the Data Scientists</h3><p>For those interested in running the script locally, feel free to clone the <a href="https://github.com/NadimKawwa/DOTAWardFinder">GitHub repo</a>, specifically <a href="https://github.com/NadimKawwa/DOTAWardFinder/blob/main/07_WardCircles.ipynb">notebook #7</a>. The snippet below shows a sample run which mimics scikit-learn’s methods for familiarity.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c56c4c8cb1d3c1203117902c53b3cc81/href">https://medium.com/media/c56c4c8cb1d3c1203117902c53b3cc81/href</a></iframe><h3>Conclusion</h3><p>The first step begins by sharing something regardless of how ‘done’ it is, and hope that someone finds it useful. When it comes to designing an app, gather feedback from users and try to anticipate their needs. Moreover, I could not have taken the project this far without invaluable input from the DOTA2 community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b8ebb58315" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/design-considerations-for-a-web-app-4b8ebb58315">Design Considerations for A Web App</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where To Place Wards in DOTA2]]></title>
            <link>https://medium.com/data-science/where-to-place-wards-in-dota2-84f534b9b64e?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/84f534b9b64e</guid>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dota-2]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Sat, 02 Oct 2021 15:23:42 GMT</pubDate>
            <atom:updated>2021-10-04T20:55:14.017Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Learning From Pro Gamers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7qFkOVkzxvcyg3n_" /><figcaption>Photo by <a href="https://unsplash.com/@evgenit?utm_source=medium&amp;utm_medium=referral">Evgeni Tcherkasski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>tl;dr: buy wards and place them in strategic locations</em></p><h3>To Be A Support Hero in DOTA</h3><p>DOTA2 is a MOBA game with many interesting and sometimes hidden mechanics. It’s a team sport and each team member is expected to play their assigned role. For example, a support hero is expected to create breathing room for farm dependent heroes, <a href="https://dota2.fandom.com/wiki/Creep_Stacking">stack camps</a>, and purchase support items.</p><p>As I learn to play support heroes, one crucial role is map vision. Indeed, vision in the right spots can help teams secure crucial objectives. Moreover, vision creates map awareness and space for your teammates to roam or ambush (<em>gank</em>) with relative safety.</p><p>The objectives of this post are:</p><ul><li>Explore when and where do pro players make use of vision with the help of some data science tools.</li><li>Serve as a guide for the casual player and improve their gaming experience.</li><li>Infer how a paid subscription like <a href="https://www.dota2.com/plus">Dota Plus</a> makes ward suggestions.</li></ul><p>The complete code can be found here:</p><p><a href="https://github.com/NadimKawwa/DOTAWardFinder">GitHub - NadimKawwa/DOTAWardFinder: Find ward clusters in DOTA</a></p><h3>Defining Vision</h3><p>The first kind are <a href="https://dota2.fandom.com/wiki/Observer_Ward">observer wards</a> that provide vision over an area. Their vision is hindered by the tree line, and an observer ward placed atop a hill provides flying vision. Note that observer wards do not reveal invisible units.</p><p><a href="https://dota2.fandom.com/wiki/Sentry_Ward">Sentry wards</a> on the other hand provide no vision of their own. However, a player standing next to a sentry ward can see invisible units.</p><p>Both kinds of wards are invisible to enemies unless a sentry ward is planted. The act of planting wards in strategic locations is called <strong><em>warding</em></strong>, and removing wards placed by the enemy team is called <strong><em>dewarding</em></strong>.</p><h3>Map Overview</h3><p>The image below shows the full DOTA map with the Radiant team in the bottom left and Dire team on the top right. For more details, see this <a href="https://dota2.fandom.com/wiki/Map">Wikia Article</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fk1IMuCOrvAWrCBC" /><figcaption>Source: <a href="https://dota2.fandom.com/wiki/Map">https://dota2.fandom.com/wiki/Map</a> Under <a href="https://www.fandom.com/licensing">CC BY-NC-SA 3.0</a> License</figcaption></figure><p>The <a href="http://opendota.com/">OpenDota</a> project allows players to view a summary of matches and useful plots like the one below taken from a <a href="https://www.opendota.com/matches/6169617852/vision">sample match</a>:</p><ul><li>Green is Radiant</li><li>Red is Dire</li><li>Oval with iris is an observer ward</li><li>Oval with no iris is a sentry ward</li></ul><p>Note that observer wards have a much larger line of sight and that the map does not account to hindrances due to trees or high ground.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*b7M75JjD65PbRN-T6ioryA.png" /><figcaption>Sample ward map by <a href="https://www.opendota.com/matches/6169617852/vision">OpenDota </a>available under MIT License</figcaption></figure><h3>Data Collection</h3><p>The data is free to query for anyone from OpenDota:</p><p><a href="https://www.opendota.com/explorer">Data Explorer - Professional Dota 2 Stats - OpenDota - Dota 2 Statistics</a></p><h4>SQL for Parsing Data</h4><p>The following SQL query returns the log data from professional matches. The time frame is arbitrarily set to January 1st 2021 and beyond. In addition, 25000 matches ought to suffice for our purposes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d4c45646b83153cbc269398f0275ea8/href">https://medium.com/media/3d4c45646b83153cbc269398f0275ea8/href</a></iframe><h4>Data Format</h4><p>The image below shows the two main columns of interest. The logs are in JSON format and will be flattened for better data ingestion. The following info matters for this project:</p><ul><li>(x,y) location of ward.</li><li>Sentry or observer.</li><li>Time placed.</li><li>If placed by Radiant or Dire team.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Y2Lv7N-9XUd7tUR1ZfRdw.png" /><figcaption>Screengrab by Author</figcaption></figure><h4>OpenDota Coordinate System</h4><p>OpenDota uses a coordinate system different from the actual map size. Indeed, the lower left corner of the map corresponds to coordinates (64,64) and the map is divided into quadrants that are numbered from 0 to 127. Hence, the upper right corner of the map has coordinates (192, 192).</p><h3>Data At A Glance</h3><p>How does the data look? The image below shows a scatter of all the observer wards in the dataset, and map coloring by height. The blue points (z=132) correspond to a hill, such that a ward placed atop a hill provides unobstructed vision. In addition, a red bounding box is added to show the extent of the map.</p><p>Fun fact: the empty gray spot at (100,150) corresponds to <a href="https://dota2.fandom.com/wiki/Roshan">Roshan’s Pit</a> and no wards can be placed inside of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*dpVHV5qb03_KJ2GkWvcmLg.png" /><figcaption>Plot by author</figcaption></figure><p>The previous plot is very messy and doesn’t reveal much. Therefore, let’s look at all the wards placed over time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*ei5BhfOKH-sS7nL50SuNMA.png" /><figcaption>Plot by author</figcaption></figure><p>One might ask: Is the warding activity the same for both teams? The plot below confirms that yes, the activity is the same regardless of team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*h27u0l0-Pwhlk_bTQe8h_Q.png" /><figcaption>Plot by author</figcaption></figure><p>In addition, let’s look at the warding activity for hills in the plot below. It follows the same trend as the overall warding activity and falls off after minute 40 of the game.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*P4V1FnoLj9FbP8axFmuz3A.png" /><figcaption>Plot by author</figcaption></figure><h3>How To Identify Ward Clusters</h3><p>Consider the plot below which shows all observer wards placed before the game begins. At a glance we can infer where players would want to plant a ward, but not where they prioritize doing so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*p3CW3D9__JcDpPOl51LaGw.png" /><figcaption>Plot by author</figcaption></figure><p>What if we wanted to identify locations with heavily contested vision? Where do teams prioritize warding? We can answer these questions using the clustering algorithms explained in the next section.</p><h4>Model of Choice: DBSCAN</h4><p><a href="https://scikit-learn.org/stable/modules/clustering.html#dbscan">DBSCAN</a> is governed by two crucial parameters:</p><ul><li>epsilon: The maximum distance between two samples for one to be considered as in the neighborhood of the other.</li><li>min_samples: The number of samples in a neighborhood for a point to be considered as a core point.</li></ul><p>A benefit of DBSCAN is that it does not require one to specify the number of clusters and it can find arbitrarily shaped clusters. Moreover, it is has a notion of noise and is robust to outliers. Therefore, some wards will not be assigned to a cluster and left as outliers.</p><h4>Why Not Use KMeans?</h4><p><a href="https://scikit-learn.org/stable/modules/clustering.html#k-means">KMeans</a> could be used as an alternative clustering algorithm. However, it relies on three assumptions:</p><ul><li>The variance of the distribution of each variable is spherical (KMeans likes spherical blobs).</li><li>All variables have same variance.</li><li>Each cluster has roughly equal number of observations.</li></ul><p>It’s the third assumption that most severely hinders KMeans. Indeed, some locations are fine with a few wards here and there such as the mid lane, and others are constantly contested such as the safe lane.</p><h4>Influence of Time: Late Game Vs Early Game</h4><p>DOTA is a game of seconds and every moment is a chance to make something new happen. Indeed, the influence of time (stage of the game) is a major factor for warding strategies.</p><p>It doesn’t make sense to place a sentry ward in your base if the tier 1 tower is still standing. Hence, it’s more insightful to cluster wards based on arbitrary time frames than to ignore the influence of time.</p><h3>Clusters of Map Vision At Each Stage</h3><p>This section contains plots of warding divided by sentry/observer and Radiant/Dire. In addition, a commentary is provided to explain how the clusters reflect the players’ strategy.</p><h4>Pre-Game — Before Minute 0:00</h4><p>Before the first creep waves leave the base, players compete for gold runes. Therefore, players place observer wards near the center of the map at the river in order to see what the enemy is doing. Moreover, some place observer wards deep in the enemy jungle before the enemy has time to deward it.</p><p>Lastly, the sentry wards shown in red correspond to economical warfare. Indeed, a sentry ward placed inside the bounding box of a neutral creep camp prevents the latter from spawning. Hence the enemy team is denied gold and the ability to pull the creep wave.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5i3ur39H0RrlkxwzrXzNQ.png" /></figure><h4>Early Game — 0:00 to 10:00</h4><p>In the early game the mid lane and the safe jungle are still contested. Since tier 1 towers are probably still standing, priority vision is given near the jungle entrances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vtr03O6ouEjHQUdE3D-2nw.png" /></figure><h4>Early to Mid Game — 10:00 to 20:00</h4><p>In the early to mid game, both dire and radiant attempt to place observer wards deep in the enemy’s safe jungle. At this stage of the game, the triangle starts becoming a contested area.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*27eZXxH_Fc1Ti8m7xkXSRg.png" /></figure><h4>Mid Game — 20:00 to 30:00</h4><p>Vision is mostly concentred at the triangles and on hilltops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rl9PUaEZwfE1xIbzFJt29Q.png" /></figure><h4>Mid to Late Game — 30:00 to 40:00</h4><p>In the mid to late game, teams can turn their attention to Roshan’s pit. This is represented by the hilltops near the pit being populated with obsever wards and sentry wards placed at the entrance to detect an attempt to snatch the aegis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEAoTgdt6QO3fPJDKfE2gw.png" /></figure><h4>Late Game — 40:00 to 60:00</h4><p>At this stage, Roshan’s pit is still a relevant objective. Emphasis is given to the triangles. Some sentry wards are placed near the base.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z3R5LPZjKQxSEmpDSoEFMA.png" /></figure><h4>Very Late Game — 60:00 and Beyond</h4><p>In this very late stage of the game, the barracks and tier 3 towers are probably gone. In order to make up for the loss of the tower’s vision, wards are placed inside the base.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1w5FYCVbD8F1QfZi64pQWQ.png" /></figure><h3>Creating An Interactive Explorer</h3><p>The inputs and parameters for the plots are arbitrary and based on domain knowledge (i.e. hundreds of hours gaming). If you wish to tweak with the data, please play around with the streamlit app below:</p><p><a href="https://share.streamlit.io/nadimkawwa/dotawardfinder/main/appWardFinder.py">https://share.streamlit.io/nadimkawwa/dotawardfinder/main/appWardFinder.py</a></p><p>The screenshot below shows the app and asks the user for 3 inputs:</p><ul><li>Value of epsilon</li><li>Time frame</li><li>Minimum samplers per cluster</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*YsbtXRrYXhFlOZkKceqzuQ.png" /></figure><h3>Advantages of This Project</h3><p>This project uses real and reliable data at a massive scale. It is therefore likely to capture the best practices of pro players and keep up with changes in strategy and game mechanics.</p><p>Moreover, the method used here is agnostic to changes in the map. Hence, any change to the map layout or elevations are already accounted for in the output.</p><h3>Inherent Limitations</h3><p>This project does not evaluate the lifetime (survival) of a ward. Players are confronted with dueling priorities:</p><ul><li>Ward an area the enemy might deward because it provides advantageous vision while it’s there.</li><li>Ward a lesser known spot, lose out on key vision but enemy less likely to deward it.</li></ul><p>Furthermore, a sentry ward costs 50 gold and provides no bounty if destroyed. On the other hand, an observer ward is free but provides 100 gold bounty and an extra 1 gold for each 15 seconds of its lifetime. Hence there is a cost/benefit ratio aspect to warding that this post does not address.</p><h3>Conclusion</h3><p>It is possible to query data and identify key map locations where wards can help teams secure kills or key objectives. The advantage of this post’s methodology is that it keeps up with changes in map layout and gameplay mechanics. Hence by observing a large amount of matches, we can discover key locations and implement them in our gameplay .</p><h3>References</h3><ul><li><a href="https://github.com/odota">The OpenDota Project</a></li><li><a href="http://varianceexplained.org/r/kmeans-free-lunch/">K-means clustering is not a free lunch</a></li><li><a href="https://en.wikipedia.org/wiki/DBSCAN#Disadvantages">DBSCAN - Wikipedia</a></li><li><a href="https://github.com/odota/core/issues/1473">Hero death coordinate · Issue #1473 · odota/core</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84f534b9b64e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/where-to-place-wards-in-dota2-84f534b9b64e">Where To Place Wards in DOTA2</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Do A/B Testing On MOBA Games]]></title>
            <link>https://medium.com/data-science/how-to-do-a-b-testing-on-moba-games-e44891c5701?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/e44891c5701</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[dota-2]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Tue, 07 Sep 2021 18:41:04 GMT</pubDate>
            <atom:updated>2021-09-07T18:41:04.997Z</atom:updated>
            <content:encoded><![CDATA[<h4>An Example With Pudge the Butcher</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E95NJB7F9bad_m3u" /><figcaption>Photo by <a href="https://unsplash.com/@has__sky?utm_source=medium&amp;utm_medium=referral">Vladislav Bychkov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>DOTA2 is a popular Massive Online Battle Arena (MOBA) game with millions of daily active users. Players pick from 121 heroes and play in 5v5 games. The game is known to have a steep learning curve and there is no one hero that guarantees a win. Indeed, for every hero there seems to be another that is the perfect counter to it.</p><p>DOTA2 is also a constantly evolving game where items and heroes get either <strong><em>buffed </em></strong>or <strong><em>nerfed </em></strong>in order to keep the game balanced. For example, patch 7.30 was recently released and the community was quick to point out heroes that are now “<em>broken</em>”, i.e. buffed so much they have a ridiculously high win rate (<a href="https://www.youtube.com/watch?v=tp-BAkXv9XQ">see this example</a>).</p><p>This raises the question: How can DOTA2 developers keep the game balanced? How do we determine if a hero was successfully buffed/nerfed?</p><p>The purpose of this article is to show how hypothesis testing can serve as a measure of game balancing, and how to interact with the OpenDOTA API.</p><h3>Hypothesis Formulation</h3><p>The hero of choice for this post is <a href="https://www.dota2.com/hero/pudge">Pudge</a>, one of the <a href="https://www.dotabuff.com/heroes/winning?date=all">most picked</a> and versatile DOTA2 heroes. It’s a hero that has been nerfed/buffed very often and elicits some strong reactions from the DOTA2 community (<a href="https://www.reddit.com/r/DotA2/comments/92n7gw/nerf_pudge_already/">see an example here</a>). Pudge also has its fair share of controversy due to the <a href="https://www.youtube.com/watch?v=d6H-HEpnlk8">fountain hook glitch</a> which gave the hero a guaranteed kill if paired with <a href="https://www.dota2.com/hero/chen">Chen</a>.</p><p>Was Pudge nerfed? We need to look at the change in win rate defined by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*NAd28swdvpGBODDzPgkdPA.png" /></figure><p>It’s useful to think of the odds of winning a game as a coin toss: you either win or lose. We are therefore dealing with a binomial distribution, and if the sample size is large enough (typically n&gt;30), we can use a z-test. For this test we use a two-tailed test with P-value of 0.05 and a corresponding z score of 1.96.</p><p>The hypotheses are:</p><ul><li><strong>Null Hypothesis:</strong> 2019 Pudge is no different from 2020 Pudge, the win rate is the same.</li><li><strong>Alternate Hypothesis</strong>: 2019 Pudge is different from 2020 pudge, the win rate has changed.</li></ul><p>We are comparing means and can use the formula below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*IYf6XR32cU5efcB5LNfyAg.png" /></figure><p>such that p_hat is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/101/1*GbnQbJXeEYpGeb1cG12H7g.png" /></figure><h3>Data Collection</h3><p>The data is obtained using the <a href="https://www.opendota.com/explorer?sql=select%20match_id%20from%20matches%0Ajoin%20team_match%20using(match_id)%0Awhere%20team_id%20%3D%2036%0Aorder%20by%20match_id%20desc">OpenDota Explorer</a>, which allows us to write SQL queries to extract data. The full table schema can be found here:</p><p><a href="https://github.com/odota/core/blob/master/sql/create_tables.sql">core/create_tables.sql at master · odota/core</a></p><h4>Data Schema</h4><p>Obviously we won’t query the entire database, the figure below shows the tables of interest and relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*Ab5rPg7H-hNp1WkrLy62pA.png" /><figcaption>Sketch by the author</figcaption></figure><h4>Implementation in SQL</h4><p>As an example, we can extract 2020 data using the query below. Note that although we want matches for the year 2020 the time frame is arbitrarily limited by major patches (see lines 17–20).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81c9c85db14aad886fa5d5eb7580dad9/href">https://medium.com/media/81c9c85db14aad886fa5d5eb7580dad9/href</a></iframe><p>The result is the snapshot below which we can download as a csv table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BgOV6gOAnXL30QwughnsA.png" /><figcaption>Screengrab by author</figcaption></figure><h3>Experiment Results</h3><p>The resultant p value for comparing win rates is 0.570 and indicates that we should retain the null hypothesis and reject the alternative hypothesis.</p><p>However, one must ask: Are we comparing apples to apples? One way to assess that is by evaluating additional variables.</p><h3>Match Duration</h3><p>One control variable we can look at is the match duration. The z score for the difference in means is 2.754 and indicates that the match durations are statistically different between both samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*j7Eul0tdAtXSoOhKzGknbA.png" /><figcaption>Image by Author</figcaption></figure><h3>Team Choice</h3><p>Is the choice of team Radiant/Dire the same for both 2019 and 2020 samples? The p-value for this experiment is 0.007, therefore yes the choice of team is the same for both time periods.</p><h3>Limitations</h3><p>The first limitation is the number of samples. Indeed, the query returned around 250 games for 2020 and 650 for 2019. Compare that to the <a href="https://www.oneesports.gg/dota2/dota-2-now-has-over-5b-games-played/">average 2.2 million matches played daily</a> and the sample size looks underwhelming.</p><p>Match duration is another variable that is more complex than its face value. Indeed, each hero has a different power spike that is a function of game duration. For example, some heroes are extremely powerful in the early game and fall off after the 30th minute.</p><p>Moreover, the matches returned are from players who consent to make their game data public. More specifically, the matches tend to be from professional players and league games. Hence, the experiment results can’t be extrapolated to every single level of play. This limitation can easily be remedied if the player rating (<a href="https://dota2.fandom.com/wiki/Matchmaking_Rating">MMR</a>) is known.</p><h3>Conclusion</h3><p>The experiment ended up inconclusive as we don’t know for sure if Pudge was nerfed or buffed. DOTA is a game with complex mechanics and although it’s frustrating not to have definite answers, it’s always fun to learn something new.</p><p>If you wish to explore the entire code, see the repository below:</p><p><a href="https://github.com/NadimKawwa/dota_hypothesis_test">GitHub - NadimKawwa/dota_hypothesis_test: hypothesis testing for win rate using a sample hero</a></p><h3>References</h3><ul><li><a href="https://en.wikipedia.org/wiki/Statistical_hypothesis_testing">Statistical hypothesis testing - Wikipedia</a></li><li><a href="https://stats.stackexchange.com/questions/113602/test-if-two-binomial-distributions-are-statistically-different-from-each-other">Test if two binomial distributions are statistically different from each other</a></li><li><a href="https://github.com/odota/core/blob/master/sql/create_tables.sql">core/create_tables.sql at master · odota/core</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e44891c5701" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-do-a-b-testing-on-moba-games-e44891c5701">How To Do A/B Testing On MOBA Games</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can I Use PCA for Feature Selection?]]></title>
            <link>https://medium.com/@nadimkawwa/can-i-use-pca-for-feature-selection-81565e726b3f?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/81565e726b3f</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Sat, 21 Aug 2021 01:39:21 GMT</pubDate>
            <atom:updated>2021-08-21T01:39:21.149Z</atom:updated>
            <content:encoded><![CDATA[<h4>Addressing Common Misconceptions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t0mOLUR6lUXLHEWV" /><figcaption>Photo by <a href="https://unsplash.com/@lunarts?utm_source=medium&amp;utm_medium=referral">Volodymyr Hryshchenko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Principal Component Analysis (PCA) is a feature reduction method often used in clustering tasks. The question arises: Can we use PCA in classification tasks? Is PCA the remedy to the curse of dimensionality?</p><h3>PCA for Classification</h3><p>This is a trap: we should not use PCA in classification tasks.</p><p>PCA will find the most accurate data representation in a lower dimensional space. However, the direction of maximum variance may be useless for classification.</p><p>To provide a practical example, consider a two dimensional toy dataset with two classes:</p><ul><li>c1=[(1,2),(2,3),(3,3),(4,5),(5,5)]</li><li>c2=[(1,0),(2,1),(3,1),(3,2),(5,3),(6,5)]</li></ul><p>Plot the data in a scatter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*zkRNA7f-rHWaJoPum2hR1w.png" /></figure><p>Suppose we want to reduce the data to a single component, we get the plot below. It’s visibly clear how the reduced data makes it really hard to have a clear decision boundary. Therefore, PCA is not always guaranteed to aid in classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*-WWc9m4zyerPwkEjc-H0zw.png" /></figure><p>For a more theoretical background, I highly recommend the paper<a href="https://www.win.tue.nl/~mpechen/publications/pubs/PechenizkiyCBMS04.pdf"> PCA-based Feature Transformation for Classification: Issues in Medical Diagnostics</a>. Quoting the latter on PCA:</p><blockquote>Although PCA “spreads out” data in the new basis, and can be of great help in unsupervised learning, there is no guarantee that the new axes are consistent with the discriminatory features in a classification problem.</blockquote><h3>Fisher’s Linear Discriminant</h3><p>For feature reduction, we can instead consider Fisher’s Linear Discriminant (FLD). Imagine we have 2 classes and and d-dimensional samples (x.1, x.2, … x.n), and for simplicity let’s say we have a two-dimensional dataset.</p><p>Consider projecting an observation point xi on the blue line shown in the sketch below. The line direction is given by the unit vector v. Therefore, the scalar vT xi is the distance of projection of xi from the origin. Hence, this vT xi is the projection of xi into a one dimensional subspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*LMPzIf2ZXEX5zMvX9cqw7Q.png" /></figure><p>If µ1 and µ2 are the means of classes 1 and 2 respectively, then we denote µ1_hat and µ2_hat as the means of the projections of classes 1 and 2. In mathematical notation this is expressed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*P5ZUZmLN8PqwzmQ9u0OrZw.png" /></figure><p>What if we used the absolute difference of projected means as a measure of separation? That is, what if we maximized this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*wezrXUC5oI93W1sM_vQgIg.png" /></figure><p>The scatter plot below shows an instance of projecting the observations on either axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*UpnskNiZpLMXYTX4lEy_-w.png" /></figure><p>Clearly, the projection on the vertical axis provides a better separation of the classes. However, the difference in projected means is greater along the horizontal axis. This entails that using the absolute difference of projected means is not a good measure of class separation.</p><p>The remedy to this problem is to normalize the absolute difference by a factor proportional to variance: scatter. Scatter is simply the sample variance multiplied by the number of observations. If yi is the projected samples of xi, then the scatter for the projected samples of class1 is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*wCbAwWxXp4IYUStUpJdjxA.png" /></figure><p>Therefore FLD is to project the data points on a line in the direction v that maximizes this equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*f7TDPtDdciP3tEzNhkSLBQ.png" /></figure><ul><li>High nominator signifies that the means are far from each other.</li><li>Small s1_hat signifies that samples of class <strong>1</strong> are clustered around the projected mean µ1_hat .</li><li>Small s2_hat signifies that samples of class <strong>2</strong> are clustered around the projected mean µ2_hat .</li></ul><p>Let S1 and S2 denote the scatter matrices of the original samples of classes 1 and 2 respectively. We can write S1 as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*uzwaeQIt_fdrsB14KLumig.png" /></figure><p>In addition, define the <strong>within </strong>class scatter matrix SW and the <strong>between </strong>class scatter matrix SB as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*IHkp_Ete2cKyPg-5HOx3Tg.png" /></figure><p>Going back to J(v) derived with respect to v, we get a generalized eigenvalue problem and a neat final formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*DsmJq7K3L3wX_1ZpCvmHpA.png" /></figure><h4>Python Implementation</h4><p>Below is an implementation of the math. For a more thorough implementation, see <a href="https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html">sklearn</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b15b016e4577d612ac0b47d63f08e1b4/href">https://medium.com/media/b15b016e4577d612ac0b47d63f08e1b4/href</a></iframe><h3>References</h3><p>This article is based on a lecture from University of Waterloo found here: <a href="https://www.csd.uwo.ca/~oveksler/Courses/CS434a_541a/Lecture8.pdf">https://www.csd.uwo.ca/~oveksler/Courses/CS434a_541a/Lecture8.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81565e726b3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting Malware in Android Apps]]></title>
            <link>https://medium.com/data-science/detecting-malware-in-android-apps-a011d84b931b?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/a011d84b931b</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[android]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Sat, 17 Jul 2021 16:53:11 GMT</pubDate>
            <atom:updated>2021-07-17T16:53:11.038Z</atom:updated>
            <content:encoded><![CDATA[<h4>Examining Android App Stores for Malicious Content</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VFKVFKtVZSmyT15i" /><figcaption>Photo by <a href="https://unsplash.com/@redaquamedia?utm_source=medium&amp;utm_medium=referral">Denny Müller</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong><em>TL;DR</em></strong><em>: Bad guys abuse permissions and outdated software to infect your devices.</em></p><h3>Background: Epic Games, Inc. v. Apple Inc.</h3><p>At the time of writing this piece, <em>Apple Inc.</em> and <em>Epic Games, Inc.</em> are in the throes of a <a href="https://cand.uscourts.gov/cases-e-filing/cases-of-interest/epic-games-inc-v-apple-inc/">legal dispute</a>. Epic Games claims that Apple’s App Store is a monopoly, and should not have control over in-app purchases. Apple countersued Epic and accused it of breaching its contract. The outcome of the ruling could fundamentally alter how Apple runs its App Store and how app developers monetize their content.</p><p>There is one aspect to the lawsuit that piqued my interest: Apple’s argument for maintaining its current model of having complete control over the App Store.</p><p>The argument goes that Apple’s tight grip over the App Store guarantees a safe environment where every single application is vetted for malware and compliance with Apple’s standards. Hence, the user is shielded from malware and can focus on what’s important: finding an app you like. The metaphor of the walled garden is often used here since you can pick and enjoy any fruit within the garden as long as you stick within the confines of its walls.</p><p>This got me asking: What’s the security landscape in a third party store compared to Apple’s? Can we create a simple data-driven model to vet apps for malware regardless of who controls the store?</p><p><em>Full disclosure: This entire project was done on an Apple MacBook, I’m not taking any sides in the legal case.</em></p><h3>Objectives</h3><p>My objective in this piece is to show how a data-driven approach can flag malicious smartphone apps, and provide an explanation for each step leading up to that goal. I will also see how my results compare to other research on the topic.</p><p>The operating system (OS) examined is <a href="https://en.wikipedia.org/wiki/Android_(operating_system)">Android</a>, which has a 72% share of the world wide mobile OS market. Given its widespread popularity, it is more likely to be abused by malicious actors.</p><h3>Show Us the Code</h3><p>The repository associated with this project can be found here:</p><p><a href="https://github.com/NadimKawwa/AndroidMalware">NadimKawwa/AndroidMalware</a></p><h3>Getting the Data</h3><p>The data is from Kaggle user <a href="https://www.kaggle.com/saurabhshahane">Saurabh Shahane</a>, and can be downloaded in one click:</p><p><a href="https://www.kaggle.com/saurabhshahane/android-malware-dataset">Android Malware Dataset</a></p><p>The dataset was originally used in the paper “<em>ADROIT: Android malware detection using meta-information” </em>by<em> Martín, Alejandro; Calleja, Alejandro; Menéndez, Héctor D.; Tapiador, Juan; Camacho, David</em>.</p><p>Each row in the dataset represents meta-information of an app available on the <a href="https://en.aptoide.com/">Aptoide app store website</a> and also information from the app’s Android Manifest. In addition, the paper will serve as a benchmark for model performance.</p><p>Please note that I assume that the information in the dataset is as truthful and accurate as possible. Any changes to this fundamental assumption might entirely alter the findings of this article. In addition, I highly recommend reading the paper as it explains how the data was aggregated and labeled.</p><h3>Building A Feature Space</h3><p>In this section, I give a general description of how I munge disparate data types into actionable features.</p><h4>Target Variable &amp; Success Metrics</h4><p>Rather straightforward, the target variable takes one of two values: benign or malware. The data is about 70% benign and 30% malware. For comparison, according to Apple CEO Tim Cook, only 1–2% of the Apple App Store is malware.</p><p>The authors of the ADROIT paper achieved 94% AUC. I will see if I can replicate similar results with my methods.</p><p>Moreover, I will also use F1 score for checking model performance. Indeed, ROC averages over all possible thresholds whereas the F1 score applies to any arbitrary point on the ROC curve.</p><p>Using the F1 score also raises core business questions: do we sacrifice precision for more recall? Is the cost of missing a malware app much higher than barring well-meaning developers from using the marketplace (False Positive)?</p><p>To rephrase the previous question: For a store like Aptoide, how much potential income would they be missing out on if they barred a well-meaning developer? On the other hand, how much reputational damage, user harm, and lost income is there from missing malware?</p><p>Furthermore, not all malware is created equal. Indeed, it can be adware, spyware, ransomware, or some other kind.</p><p>These questions are tied to core business benchmarks and are beyond the scope of this article.</p><h4>Numerical Features</h4><p>Out of 166 numerical features, 160 are binary and 6 are continuous. We have too many features on our hands, and it might hinder a model’s predictive performance. For this reason, it’s useful to look at each feature’s variance and remove features that have a low variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*za40F93zHi62O01giNdpkA.png" /></figure><p>For the continuous variables, we have to look at outliers. This is particularly relevant to the number of ratings an app receives since an overwhelming number of apps never received a single rating.</p><p>In order to weed out outliers, I used <a href="https://www.stat.cmu.edu/~cshalizi/statcomp/13/labs/05/lab-05.pdf">Tukey’s method</a>: the outliers are values more than 1.5 times the interquartile range from the quartiles, either below <em>Q1 − 1.5IQR</em>, or above <em>Q3 + 1.5IQR</em>.</p><h4>Text Features</h4><p>When it comes to NLP, I usually pose the following questions:</p><ul><li>Can I understand what a column is about by skimming the text?</li><li>Does it use the Latin alphabet and is it all in the same language?</li><li>What are the preprocessing functions needed?</li><li>What’s the most compact and representative way to transform the text?</li></ul><p>As it turns out, many entries have a description in Chinese characters (Also some Turkish, Korean, etc…). If one day I want to productionize or explain this model, I will have to understand and explain its choices to stakeholders. However, I can’t explain that which I don’t understand and therefore decided to drop non-Latin characters from the text.</p><p>Below is a sample from an app description:</p><pre>Tap Tap Gems is a very addictive puzzle game.  Tap two or more gems with the same colour to remove them from the board.  You must think which gems should be removed next.  To pass each level you must score certain amount of points.   The more gems removed the more points you score.  Game features 120 levels and 7 different gems.</pre><p>And another sample shows that even with strictly Latin characters, one might still have no idea what is going on (I’m guessing this is Vietnamese?).</p><pre>Game bi dn gian Vit bao gm 15+ cc mini quen thuc vi ngi Vit ta t xa n nay nh: Xm, Phm, Tin ln min Nam, Ling, Poker, X t, Mu binh, Bi co, X dzach v Tin ln min Nam m l.  t lu Tin Ln Min Nam m L - Game bi dn gian Vit  c cc game th yu qu bi  ha cc p, giao din c chm sc rt k. Cc chc nng trong game hn hn cc game cng loi khc nh</pre><p>After preprocessing, the text data is transformed using TF-IDF vectorization. In order to optimize prediction, TF-IDF parameters are tuned using a grid search cross validation.</p><h3>Model Selection</h3><p>At first I wanted to choose XGBoost since it had the highest AUC. So I set out to do a vanilla data science workflow that comes after data munging:</p><ul><li>Instantiate TF-IDF vectorizer</li><li>Instantiate a pipeline to avoid leakage</li><li>Research how parameters influence the model predictions with GridSearchCV</li></ul><p>With an AUC of 0.94, I was happy since it matched the authors’ findings. However, when comparing train/test AUC, the plot below suggested I was overfitting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*nL0IKs8JdwkHl8x26nqaoA.png" /></figure><p>In contrast, Random Forest (RF) shows underfitting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*2GEXasu1cfguFqP9JCu2QQ.png" /></figure><p>Logistic regression is not overfitting, but it might be severely underfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*xGWidq6wScfB7zHkM73Qhw.png" /></figure><p>XGBoost should be the clear winner here, however, I decided to continue with a vanilla logistic regression. It comes with a rather straightforward interpretability and saves me the trouble of using methods such as <a href="https://christophm.github.io/interpretable-ml-book/shap.html">SHAP</a>, <a href="https://christophm.github.io/interpretable-ml-book/feature-importance.html">feature permutation</a>, or <a href="https://christophm.github.io/interpretable-ml-book/lime.html">LIME</a>.</p><h3>Effect of Corpus Size on Predictions</h3><p>Since this task involves NLP, I want to mine as much information as possible from the text data. Logistic regression tends to suffer when working with sparse data, and sparse data is typical when working with text.</p><p>We have two plots below, one is the ROC curve and the other is the precision recall curve. Multiple curves show how as we add more features the model performance improves only slightly.</p><p><strong>Spoiler Alert</strong>: Text data did not matter, the next section will show that the numerical features in the Android Manifest are the most important factors for classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DApVgWAExKjgsWLlRXKx_w.png" /></figure><h3>Interpretation of Results</h3><p>It’s fair to assume that black box models are not exactly popular. Therefore, I set out to understand how the classifier weighs its decisions and dive deeper into these features.</p><h4>Indicators FOR Malware</h4><p>In terms of what is a strong indicator for malware, the top 5 are:</p><ol><li>android.permission.READ_SYNC_SETTINGS</li><li>android.permission.WRITE_SYNC_SETTINGS</li><li>android.permission.MANAGE_ACCOUNTS</li><li>com.android.launcher.permission.UNINSTALL_SHORTCUT</li><li>android.permission.RECEIVE_BOOT_COMPLETED</li></ol><p>Take for example the permission to <a href="https://developer.android.com/reference/kotlin/android/Manifest.permission#write_sync_settings">write sync settings</a>, what does it mean? When a device synchronizes (<em>syncs</em>), it is synchronizes data on your phone with the service’s servers.</p><p>The table below is a snapshot from the training data, and it suggests that giving an unvetted app these permissions is a strong case for caution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*iF_bhMfYmU3dHtFHJiSYVg.png" /></figure><p>In addition, <a href="http://androidpermissions.com/permission/android.permission.MANAGE_ACCOUNTS">MANAGE_ACCOUNTS</a>, allows the app to add/remove accounts and delete their passwords.</p><p>Moreover, the <a href="https://developer.android.com/reference/android/Manifest.permission#RECEIVE_BOOT_COMPLETED">RECEIVE_BOOT_COMPLETED</a> stood out as a strong indication of compromise. The android documentation sums it up nicely:</p><blockquote>Though holding this permission does not have any security implications, it can have a negative impact on the user experience by increasing the amount of time it takes the system to start and allowing applications to have themselves running without the user being aware of them.</blockquote><p>Lastly, the <a href="https://developer.android.com/reference/kotlin/android/Manifest.permission#uninstall_shortcut">UNINSTALL_SHORTCUT</a> permission is no longer supported. This might be associated with malware as older software tends to have more outstanding vulnerabilities.</p><h4>Indicators AGAINST Malware</h4><p>For the indicators against malware, the main features are not as obvious:</p><ol><li>android.permission.BLUETOOTH</li><li>android.permission.READ_LOGS</li><li>.//Min_SDK</li><li>android.permission.READ_EXTERNAL_STORAGE</li><li>“ringtones”</li></ol><p>It’s not entirely clear to me why permissions to connect to paired <a href="https://developer.android.com/guide/topics/connectivity/bluetooth">Bluetooth</a> and reading low-level system <a href="https://developer.android.com/reference/android/Manifest.permission#READ_LOGS">log files </a>is a case against malware. Perhaps it’s not as bad as other permissions that allow the device to advertise to nearby Bluetooth devices (see <a href="https://developer.android.com/reference/android/Manifest.permission#BLUETOOTH_ADVERTISE">BLUETOOTH_ADVERTISE</a>).</p><p>Moreover, it’s beyond me why having “ringtones” in the app description gives you the green light to download it.</p><p>However, I can comprehend how setting a higher SDK version (<a href="https://developer.android.com/guide/topics/manifest/uses-sdk-element">Min_SDK</a>) can protect your device against infection. This feature aligns well with <a href="https://developer.android.com/reference/android/Manifest.permission#READ_EXTERNAL_STORAGE">READ_EXTERNAL_STORAGE</a>. The latter is enforced starting API level 19 and prompts the user for themselves to enable/disable the permission.</p><p>Unfortunately, the referenced paper does not come with code or an explanation of model weights. Therefore, I’m unable to rigorously compare my findings with the authors’.</p><h3>Conclusion</h3><p>Working with data from the Aptoide store suggests that there’s an ample amount of malware on third party stores. Therefore, Apple’s business practices could seem reasonable if they guarantee the safety of the user.</p><p>However, data science can be leveraged to create a speedy and accurate classification system where such restrictions by Apple are not necessary. The logistic regression model is simple to set up and offers adequate performance when compared to existing literature.</p><p>Projects like these have the potential to boost the safety of app stores and give a deeper understanding of how vulnerabilities are exploited. The methodology can be expanded to other stores and operating systems.</p><h3>Before You Go</h3><p>If you’re interested in learning more about the intersection between cybersecurity and data science, check out this other project:</p><p><a href="https://towardsdatascience.com/cyber-security-threat-identification-2affad0989c">Cyber Security Threat Identification</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a011d84b931b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/detecting-malware-in-android-apps-a011d84b931b">Detecting Malware in Android Apps</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Reddit Took Over Wall Street]]></title>
            <link>https://medium.datadriveninvestor.com/how-reddit-took-over-wall-street-e830ba2b5848?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/e830ba2b5848</guid>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[reddit]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Tue, 02 Mar 2021 04:00:50 GMT</pubDate>
            <atom:updated>2021-03-20T13:02:06.501Z</atom:updated>
            <content:encoded><![CDATA[<h4>Can’t Stop Won’t Stop GameStop 🚀🚀🚀</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*koTnXgNUVZ9iTkru" /><figcaption>Photo by <a href="https://unsplash.com/@jlondonbaker?utm_source=medium&amp;utm_medium=referral">John Baker</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>On January 28th 2021, the price of GameStop Corp. (<a href="https://finance.yahoo.com/quote/GME/">$GME</a>) shares shot up to an all time high of $483, an approximate 100 fold increase compared to its listing price a year ago. As news outlets scrambled to obtain an explanation, a subreddit called r/WallStreetBets (WSB) was singled out as the driving force behind the surge.</p><p>If you go on WSB, you will mostly see a whole lot of rocket emojis (🚀🚀 🚀 🚀 🚀), diamond hands(💠 🙌), paper hands (🧻 🙌), “Apes together strong 🦍” , talk of “tendies” 🐔, and “Still holding” with screenshots of portfolios deep in the red. If the slang on WSB seems confusing, <a href="https://www.reuters.com/article/us-retail-trading-slang/factbox-stonks-to-the-moon-deciphering-reddits-wallstreetbets-lingo-idUSKBN29X2VK">Reuters has got you covered</a>. To understand how Reddit itself works, refer to <a href="https://www.reddit.com/r/funny/comments/l78271/breaking_news_cnbc_solved_reddit/">CNBC’s explanation</a>.</p><p>In this article, my objective is to dissect WSB using data science. Particularly, I aim to show the trends among the swarm of loosely-coordinated, individual retail investors who <a href="https://en.wikipedia.org/wiki/Short_squeeze">short squeezed</a> GME. I will do this by first, demonstrating how to collect the data. Next, I will present plots that offer snapshots into WSB. Finally, I will use NLP to measure emotional engagement on the subreddit.</p><p><a href="https://www.datadriveninvestor.com/2020/11/26/could-blue-origin-demolish-the-spacex-monopoly/">Could Blue Origin Demolish The SpaceX Monopoly? | DataDrivenInvestor</a></p><h3>Setting up Parameters</h3><p>On July 27th 2020, youtuber Roaring Kitty (AKA u/DeepFuckingValue) <a href="https://www.youtube.com/watch?v=GZTr1-Gp74U&amp;t=88s">posted his thesis on why he’s bullish on GME</a>. This is a good starting point to begin collecting data from WSB as this is when the community probably became first aware of GME.</p><p>There are two kinds of content on Reddit: comments and submissions. A submission is an original post created by a user, whereas a comment is extracted from the thread of comments under that submission.</p><p>For each day starting July 1st 2020 until February 2021, I collected the top 100 submissions and top 100 comments in WSB in 8 hour intervals. This can be done via <a href="https://reddit-api.readthedocs.io/en/latest/#search-parameters-for-comments">pushshift</a> or more simply via <a href="https://reddit-api.readthedocs.io/en/latest/#">PMAW.</a></p><p>Creating a URL to capture that data is made easy thanks to pushshift. The python function below creates a URL that tells pushshift to collect the top 1000 posts in a time frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/342d3656e69de230d3b748c53f46d1a4/href">https://medium.com/media/342d3656e69de230d3b748c53f46d1a4/href</a></iframe><p>From that URL I can build a feature space:</p><ul><li>Time of creation</li><li>Submission title</li><li>Text content</li><li>Awards given if any</li><li>If it contains video and/or pictures</li><li>Number of comments</li><li>Flair (Meme, YOLO, Loss)</li></ul><h3>Visualizing a Subreddit</h3><p>As WSB went mainstream, users flocked to join the subreddit. What does going viral look like? The plot below shows the growth of the subreddit from July 2020 to January 2021, in what resembles a hockey stick growth. For reference, all plots contain the high and low price of GME stock on that day.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-member-count%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-member-count%2F&amp;image=https%3A%2F%2Fstorage.googleapis.com%2Fdatapane-files-prod%2Fdp%2Fthumbnails%2Fa7488141-9981-4637-aee3-7502878189fb.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="800" height="625" frameborder="0" scrolling="no"><a href="https://medium.com/media/5263eaae7206e82f99b6d8648355ea73/href">https://medium.com/media/5263eaae7206e82f99b6d8648355ea73/href</a></iframe><p>Emojis are used everywhere on the internet and WSB is no exception. The plot below shows the cumulative use of the top 10 emojis. Unsurprisingly, the rocket emoji shot up in early November and has gone up since, despite GME tanking later in January.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-emoj-cumsum%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-emoj-cumsum%2F&amp;image=https%3A%2F%2Fstorage.googleapis.com%2Fdatapane-files-prod%2Fdp%2Fthumbnails%2F7b48bc2e-b332-4fe0-b286-dd3b2c5be1ff.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="800" height="625" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6cb9ee9ea52f71c229f19977d31a15a/href">https://medium.com/media/c6cb9ee9ea52f71c229f19977d31a15a/href</a></iframe><p>Moreover, in early January 2021, it seemed like GME was the one and only topic on WSB, the plot below shows how it came to dominate conversation. The stocks of interest are based on a perusal of WSB forums and stocks associated with it. For example, Tesla and Palantir are two other WSB favorites when it comes to making bullish bets.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-ticker-cumsum%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-ticker-cumsum%2F&amp;image=https%3A%2F%2Fstorage.googleapis.com%2Fdatapane-files-prod%2Fdp%2Fthumbnails%2Fb1ad9e42-b809-4c69-a369-0e88293e1427.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="800" height="625" frameborder="0" scrolling="no"><a href="https://medium.com/media/57fa0b8459d938e2f55fb38d4b01150b/href">https://medium.com/media/57fa0b8459d938e2f55fb38d4b01150b/href</a></iframe><p>In the future though, tasks like these might become harder because <a href="https://www.reddit.com/r/wallstreetbets/comments/lnj00p/stocks_i_like_for_the_hedge_funds_buying_tech_to/">WSB users will try to throw you off with random tickers</a>.</p><h3>Online Sentiment &amp; Stock Price</h3><p>The prevailing assumption is that we live in an efficient market where assets are priced based on a logical assessment of business fundamentals. However, some economists such as <a href="https://en.wikipedia.org/wiki/Daniel_Kahneman">Daniel Kahneman </a>suggested that trades are driven by emotions such as over-confidence, fads, and arrogance.</p><p>In this section, I go through every top comment and measure the sentiment with the help of <a href="https://github.com/cjhutto/vaderSentiment">VADER</a> (<em>not the Sith Lord</em>). Bear in mind that this sentiment analysis is limited by a positive/negative scale and might not capture things like humor and sarcasm, let alone understand “GME GO BRRRRRRR 🚀”.</p><p>I wanted to capture the effect of a post that has attracted thousands of comments, and prevent it from domineering the conversation. Moreover, a post is impactful based on the amount of feedback it receives from the community. These considerations yield the function below which is applied as a multiplier on the sentiment score returned by VADER:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*d9u9QTxjYi0wzV2YnILkKA.png" /><figcaption>Source: The Author</figcaption></figure><p>My reasoning is that upvote ratio is always between 0 and 1, and a post can only trend for up to 3–4 days before Reddit removes it from the trending page. However, the number of comments and awards received can range from none to tens of thousands. Therefore, number of comments and awards are better explained as logarithmic measures.</p><p>The plot below shows the 7 day rolling average for sentiments.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-emotion-7drolling%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fu%2Fnadim%2Freports%2Fwsb-emotion-7drolling%2F&amp;image=https%3A%2F%2Fstorage.googleapis.com%2Fdatapane-files-prod%2Fdp%2Fthumbnails%2Fa716a9c8-6f27-4044-af15-f1a65a87f2a5.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="800" height="625" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7fa617688adb26d731e9cc5638d41f3/href">https://medium.com/media/c7fa617688adb26d731e9cc5638d41f3/href</a></iframe><p>Zooming in on January 2021, there appears to be no significant uptick in “positive” sentiment. Moreover, the stock price of GME does not seem to be correlated. At this point, surprisingly, rocket emojis are your best indicator of market movement.</p><p>Even though VADER is specifically attuned to sentiments expressed in social media, it still falls short. It shows that when it comes to social media text, spending some time to read the entries is your best bet to understand overall sentiment.</p><h3>Predicting The Future</h3><p>It’s tempting to try and fit a regression model to predict some future astronomical rise of an asset. In this post, I abstain from doing this based on my personal conviction that stocks, in particular ‘stonks’ which only go up, are too complex to predict by some mathematical model.</p><h3>What WSB Means for Investing</h3><p>The GME phenomenon is a watershed moment in financial markets, as individual traders realized that they can organize swarm a stock. In a way, retail investors on Reddit acted like a decentralized hedge fund, and bet against established institutions.</p><p>WSB users made their case for GME and other stocks over the years, some offered insightful deep dives, while others won us over with “ I like the stock”. In the future, would you take your investing advice from Reddit?</p><h4>Before You Go</h4><p>To see how the project was coded, please refer to the <a href="https://github.com/NadimKawwa/WSB_analysis">GitHub repository</a>. Have you also worked with Reddit’s API and got insights to share? Please let me know!</p><h4>Disclaimer</h4><p><strong>PS</strong>: Nothing in this post should be taken as financial advice. I am not making any financial recommendations. You shouldn’t listen to what I have to say about stocks and/or stonks. I don’t own any shares in GME.</p><p><strong>Gain Access to Expert View — </strong><a href="https://datadriveninvestor.com/ddi-intel"><strong>Subscribe to DDI Intel</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e830ba2b5848" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/how-reddit-took-over-wall-street-e830ba2b5848">How Reddit Took Over Wall Street</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cyber Security Threat Identification]]></title>
            <link>https://medium.com/data-science/cyber-security-threat-identification-2affad0989c?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/2affad0989c</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Tue, 08 Dec 2020 18:57:19 GMT</pubDate>
            <atom:updated>2020-12-08T18:57:19.192Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to Prioritize Software Weaknesses and Vulnerabilities</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_E4MbZ9Wf5gy_AkS" /><figcaption>Photo by <a href="https://unsplash.com/@jaqbovsky?utm_source=medium&amp;utm_medium=referral">Michał Jakubowski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>A Story of Preventable Loss</h3><p>Between May and July 2017, a breach at Equifax exposed the personal information of <a href="https://www.ftc.gov/enforcement/cases-proceedings/refunds/equifax-data-breach-settlement">147 million Americans</a>. The cause was a software vulnerability known as <a href="https://nvd.nist.gov/vuln/detail/CVE-2017-5638">CVE-2017–5638</a>, and was public knowledge since March 2017 — three months before the hack occurred.</p><p>Equifax was made aware of the vulnerability a year prior to the breach and still did not act on it. The vulnerability was so damaging that a security researcher revealed to <a href="https://www.vice.com/en/article/ne3bv7/equifax-breach-social-security-numbers-researcher-warning">Motherboard</a>:</p><blockquote>“All you had to do was put in a search term and get millions of results, just instantly — in cleartext, through a web app.”</blockquote><p>The intent here is not to single out Equifax — in the USA alone, cybercrime has cost the economy between <a href="https://www.whitehouse.gov/wp-content/uploads/2018/03/The-Cost-of-Malicious-Cyber-Activity-to-the-U.S.-Economy.pdf">$57 billion and $109 billion in 2016.</a></p><p>The good news is that as many as <a href="https://us-cert.cisa.gov/ncas/alerts/TA15-119A">85% of targeted cyberattacks are preventable</a>. However, the challenge lies herein: There are more than 150,000 known Common Vulnerabilities and Exposures (<a href="https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures">CVEs)</a>, but only <a href="https://www.zdnet.com/article/only-5-5-of-all-vulnerabilities-are-ever-exploited-in-the-wild/">5.5% are ever exploited in the wild</a>.</p><p>It is logistically impossible to patch every single vulnerability, so how can we strategically target critical cases? If there are tens of thousands of CVEs, how do we identify ones that are actively exploited?</p><p>In this post I will show how data science can be leveraged to identify critical vulnerabilities in an efficient and targeted manner. This will allow for organizations to mitigate their cyber risk and allocate resources to mending critical weaknesses while avoiding unnecessary patches.</p><p>First, I will show how to collect data from disparate sources, and how to transform it into an ingestible format. Next, I will lay out the evaluation metrics, what they mean in terms of cybersecurity, and build a classification pipeline.</p><p>For implementation, check out the <a href="https://github.com/NadimKawwa/Cyber_Threat_Identification">GitHub</a> repository.</p><h3>Data Sources</h3><p>To build a model, we need data that reliably map features to a CVE and determine if it was exploited “<em>in the wild</em>”. Cybersecurity data is not easily accessible: those who have it might be wary of sharing it with malicious actors.</p><p>To make matters more complicated, our findings are limited by the visible surface of the web that we can scan. Nevertheless, there are some public resources that can be leveraged to form a comprehensive dataset. This section presents publicly available sources used in this project.</p><h4>National Vulnerability Database</h4><p>The National Vulnerability Database (<a href="https://nvd.nist.gov/#">NVD</a>) is a US government repository of common vulnerabilities and exposures (<a href="https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures">CVE</a>). NVD also provides a Common Vulnerability Scoring System (<a href="https://nvd.nist.gov/vuln-metrics/cvss">CVSS</a>) that evaluates the severity of the vulnerability.</p><p>With over 150,000 entries, the NVD is an authoritative source of cyber activity. Each vulnerability is assigned a Common Weakness Enumeration (<a href="https://cwe.mitre.org/">CWE</a>), a naming format that allows the easy sharing of information.</p><p>Take for example <a href="https://nvd.nist.gov/vuln/detail/CVE-2019-12518">CVE-2019–12518</a> shown in the screenshot below. Each CVE will contain a description of the weakness, the CVSS score, and a summary of the attack vector. The information from NVD is used to build a repository of known CVEs, the corresponding attack vectors, and the text description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*gLG8LayO4k4WUhByBNMj7Q.png" /><figcaption><a href="https://nvd.nist.gov/vuln/detail/CVE-2019-12518">NVD Sample CVE</a></figcaption></figure><h4>Exploit Database</h4><p><a href="https://www.exploit-db.com/">Exploit-db </a>is a database maintained by Offensive Security. It lists if an exploit is verified and if it has an application. In addition, it gives the platform, type of attack, and the code snippet that supposedly executes the software hack.</p><p>Exploit-db is also CVE compliant, meaning that one can map it to NVD for a more holistic information set. Exploit-db also has papers which provide a proof of concept for a hack. We opted to exclude theoretical papers from our data and limit the scope of this project.</p><p>For example, we can find the same CVE-2019–12518 as <a href="https://www.exploit-db.com/exploits/48092">EDB 48092</a> shown in the screenshot below. For this particular CVE, we know that it is verified, has a downloadable exploit, is remote, and affects Windows. Data from Exploit-db is used to augment the NVD database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11uQa9haLLx_4SUZjfeJ8A.png" /><figcaption><a href="https://www.exploit-db.com/exploits/48092">Exploit-db Sample Entry</a></figcaption></figure><h4>Cybersecurity and Infrastructure Security Agency</h4><p>Also known as CISA, it is a US government agency and can be thought of as the nation’s risk advisor. CISA protects critical government infrastructure against compromising cyberattacks. As a result, it often releases alerts and advisories about ongoing CVE exploits and potential remediation. CISA takes on a variety of roles and was recently in the spotlight for its campaign to <a href="https://www.npr.org/2020/11/17/936003057/cisa-director-chris-krebs-fired-after-trying-to-correct-voter-fraud-disinformati">combat erroneous rumors about voter fraud</a>.</p><p>The screenshot below illustrates an <a href="https://us-cert.cisa.gov/ncas/current-activity/2020/03/10/unpatched-microsoft-exchange-servers-vulnerable-cve-2020-0688">advisory for CVE-2020–0688</a>, where CISA calls on those potentially impacted to patch the weakness. Advisories like the one below are used to make a collection of exploited CVEs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*YTl81r8yewpk7UUR-fbYNw.png" /><figcaption><a href="https://us-cert.cisa.gov/ncas/current-activity/2020/03/10/unpatched-microsoft-exchange-servers-vulnerable-cve-2020-0688">CISA Sample Advisory</a></figcaption></figure><h4>Symantec</h4><p>Symantec keeps track of attempts to exploit a known operating system or application vulnerability. When an intrusion is detected, an <a href="https://www.broadcom.com/support/security-center/attacksignatures">attack signature</a> is logged along with a description of the attack. The descriptions of each attack signature can be examined for mentions of a CVE.</p><p>The screenshot below pertains to CVE-2019–12518 from before, and <a href="https://www.broadcom.com/support/security-center/attacksignatures">an attempt to exploit it has been flagged by Symantec</a>. Information from this source is used to expand the list of exploited CVEs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*3wlBODUaQru3Ya-d_Ktw9Q.png" /><figcaption><a href="https://www.broadcom.com/support/security-center/attacksignatures">Symantec Sample Attack Signature</a></figcaption></figure><h3>Making Sense of Data Features and Munging</h3><h4>Target Variable</h4><p>We may not know all of the exploited CVEs. The validity of the target variable is dictated by the depth and rigor of search for data sources. There may be CVEs that we label as not exploited that are being exploited in the wild.</p><p>In addition, a CVE might be actively exploited in the wild but of no concern to you if it lies outside your orbit. For example, if you don’t use Adobe Flash Player then <a href="https://helpx.adobe.com/security/products/flash-player/apsa18-01.html">CVE-2018–4878 </a>should be of no concern to you. On the other hand if your application is data driven, any CVE that involves a <a href="https://en.wikipedia.org/wiki/SQL_injection">SQL injection</a> should be on your radar.</p><p>To better visualize the problem, the Venn diagram below gives a global snapshot. Note how not all verified exploits have a CVE identifier and an exploit can be verified but not used in the wild.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jXx9AZuUn_rWDOizjBSFzA.png" /><figcaption>(Drawn by Author)</figcaption></figure><h4>Model Features</h4><p>We begin with the assertion that the CVSS score was designed as a measurement of severity and not of risk. The CVSS V3 successor to V2 was designed to better reflect this intent. To put things in perspective, consider a car accident and an earthquake. Every time you get in a car, it comes with a risk for a car accident. The severity of the accident can vary from negligible to loss of life. On the other hand, an earthquake is a very rare event. However, the severity of the earthquake can be exponentially higher than a car accident.</p><p>Exploring the NVD reveals that many attack vectors have descriptive words such as none, low, or high. For example, one component of the attack vector is the level of privilege (<em>PR</em> in the vector string) and varies from none to high, based on the diagram below. When ingesting data from NVD, such categorical features are transformed into numbers on a sliding scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFlWNjZcJrF03zPhZs7T7w.png" /><figcaption>Source: <a href="https://www.first.org/cvss/user-guide">CVSS User Guide</a></figcaption></figure><p>The descriptive text for a given CVE is expected to be rather formal and free of grammatical errors based on a perusal of the data entries. Note that some entries explicitly call for the CVE to be rejected and will therefore not be used for model training and classification. Moreover, because individual words bring little context, a TFIDF vectorization with unigrams to trigrams tends to deliver better ML results.</p><p>For ExploitDB we make use of <a href="https://en.wikipedia.org/wiki/One-hot">one hot encoding</a> for the type and the platform. This potentially adds a lot of features to be inputted, however it ensures that the machine learning algorithm does not assume that higher numbers are more important.</p><h3>Success Metrics and Model Selection</h3><p>The data is heavily imbalanced, such that only 1% CVEs are exploited in the wild. That 1% is far off from the 5% reported in scientific literature, however, we are constrained by the information available to us.</p><p>This section presents the metrics for selecting a winning model, what they mean algebraically, and their interpretation in the context of cybersecurity.</p><p>I assume that the reader is familiar with the notion of a <a href="https://en.wikipedia.org/wiki/Confusion_matrix">confusion matrix </a>and its components. I will begin with <a href="https://en.wikipedia.org/wiki/Precision_and_recall">precision and recall</a>.</p><p>Precision determines how many selected items are relevant:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*D9njmvWn-Bxhx3S1kYHi1w.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Precision_and_recall">(Wikipedia Formula)</a></figcaption></figure><p>If we want better precision we need to have as few false positives as possible. Therefore, an efficient model will have high precision, and allocates time and resources only to vulnerabilities that require patching.</p><p>Recall is defined by the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*xa8T7qIFU85te_9Ltn630A.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Precision_and_recall">(Wikipedia Formula)</a></figcaption></figure><p>Recall poses the question: How many relevant items are selected? A perfect recall score of 1.0 implies no false negatives were selected, therefore our model has perfect coverage. In the context of this project, the recall determines how many vulnerabilities that should be remediated have been flagged.</p><p>Looking at these two metrics individually is insufficient since, for example, we can achieve perfect recall by flagging all the CVEs in the dataset as exploited in the wild.</p><p>Therefore, the <a href="https://en.wikipedia.org/wiki/F-score">F1 score</a> offers a better evaluation metric:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*aOYH7ulL9W4VQtzQq1YnJA.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Precision_and_recall">(Wikipedia Formula)</a></figcaption></figure><p>Moreover, the <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">ROC</a> score is another useful metric for imbalanced datasets. A high AUC score signifies a high true positivity rate and a low false positive rate.</p><p>The question here is what to prioritize? An organization with limited resources might want to prioritize efficiency — A more precise model is the clear favorite. A large firm with sufficient time and resources will prefer the conservative approach and cover all their bases —A high recall is preferable. The general F1 score uses a positive real factor β, where β is chosen such that recall is considered β times as important as precision. For the purpose of this study β is taken as 1.0.</p><p>An evaluation metric beyond the scope of this project, yet worth mentioning, is the economic and intangible impact of a vulnerability. Consider a hypothetical CVE that affects a digital wallet (e.g. Apple Pay) and reveals linked credit cards. It’s easy to make the case for prioritizing this CVE because we can quantify the loss in dollar terms.</p><p>Conversely, consider another CVE that <a href="https://arstechnica.com/gadgets/2020/12/iphone-zero-click-wi-fi-exploit-is-one-of-the-most-breathtaking-hacks-ever/">gives the attacker remote access to your entire iPhone, over WiFi, and you won’t even know about it</a>. How do we measure the privacy impact of this exploit? More importantly, how do we bake these measures of urgency and importance into our classifiers?</p><h3>Start to End Workflow</h3><p>The flow chart below summarizes the previous sections and where we want to take this project. For vectorizing the text data, I opted for a simple <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html">TF-IDF vectorization</a>, and for the ML model I found XGBoost to have the best overall performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*AlUr8FicdP_Llhv7HLhbxw.png" /><figcaption>(Drawn by Author)</figcaption></figure><p>On the topic of parameters, I wanted to find out how optimizing for precision, recall, F1, or AUC would yield different hyperparameters. These parameters are:</p><ul><li>max_df: Ignore terms that have a document frequency strictly higher than the given threshold.</li><li>min_df: Ignore terms that have a document frequency strictly lower than the given threshold.</li><li>max_features: Build a vocabulary that only considers the top max features ordered by term frequency across the corpus.</li><li>max_depth: the max depth of a tree, a higher depth makes the model more complex and is more likely to overfit.</li><li>subsample: subsample ratio of the training instances.</li></ul><p>The table below shows how the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html">cross validation grid search</a> selected parameters based on the preferred scoring strategy. Notice how recall, which translates into better coverage, is averse to overfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*4YIJov0jEZ2gZR2_zuEUMA.png" /><figcaption>(Drawn by Author)</figcaption></figure><p>Visually, we can see different PRC and ROC curves in the two plots below. Overall gains in AUC scores are negligible, however, Precision/Recall vary significantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*dbNDE9LHbk0O335PTND8Gg.png" /><figcaption>(Drawn by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*ePSEfNf675mYBF8r7JpRZg.png" /><figcaption>(Drawn by Author)</figcaption></figure><p>How reliable are the ML models though? One easy way to find out is by bootstrapping the data. Using the ML model optimized for F1 score, i.e. balanced recall and precision, we get the histogram below. 95% of the time, the <strong>precision</strong> will be in the range of <strong>(0.149, 0.234)</strong>, for <strong>recall</strong> it is <strong>(0.333, 0.55)</strong>, and for <strong>F1</strong> it is <strong>(0.180, 0.337)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*S13nHD7JPNVZuIPutG0viA.png" /><figcaption>(Drawn by Author)</figcaption></figure><p>The benefit of the above graph is that it could be used to validate hypotheses for improving the ML pipeline. For example, say I declare that <a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html">TorchText</a> will deliver superior results. To back up my claim, I need to <a href="https://en.wikipedia.org/wiki/Statistical_hypothesis_testing">test my hypothesis</a> and show that TorchText is significantly better.</p><h3>Conclusion &amp; Looking Ahead</h3><p>It’s crucial to state that the findings here might be challenged in the future as new data becomes available. This project also omits existing defense mechanisms organizations might have, and only takes a bird’s eye view of things. To put it plainly: your mileage may vary based on how secure your defenses are and on the jackpot in your safe.</p><p>Furthermore, this project does not take into account the effect of time. Indeed, some CVEs might already have an existing patch, which may or may not have been implemented. It is also possible that with more sophisticated tools and higher computing power, previous innocuous CVEs can wreak havoc on an organization.</p><p>With a rather simple workflow, critical vulnerabilities can be identified with a relatively high performance. This means that as new CVEs are released on NVD, a ML model can be used to evaluate if they are likely to be exploited in the wild.</p><h4>References</h4><p>The inspiration for this project was from the conference paper <a href="https://weis2019.econinfosec.org/wp-content/uploads/sites/6/2019/05/WEIS_2019_paper_53.pdf"><em>Improving Vulnerability Remediation Through Better Exploit Prediction</em> by Jacob et al. (2019)</a>.</p><ul><li><a href="https://www.first.org/cvss/user-guide">CVSS v4.0 User Guide</a></li><li><a href="https://www.first.org/cvss/specification-document#i5">CVSS v3.1 Specification Document</a></li><li><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7338193/">Automated CPE Labeling of CVE Summaries with Machine Learning</a></li><li><a href="https://www.exploit-db.com/">OffSec&#39;s Exploit Database Archive</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2affad0989c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/cyber-security-threat-identification-2affad0989c">Cyber Security Threat Identification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Issue Smarter Loans]]></title>
            <link>https://medium.com/data-science/how-to-issue-smarter-loans-d0eda750bed9?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0eda750bed9</guid>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[fintech]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Sat, 25 Apr 2020 03:34:31 GMT</pubDate>
            <atom:updated>2020-04-25T03:34:31.638Z</atom:updated>
            <content:encoded><![CDATA[<h4>Keep Your Users Happy And Comply With Federal Regulations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C1vM24QHhRynMy9U" /><figcaption>Photo by <a href="https://unsplash.com/@neonbrand?utm_source=medium&amp;utm_medium=referral">NeONBRAND</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Applying for a loan can be a stressful experience for a variety of reasons. In this post, we will use a worked example to understand how data science is leveraged to issue better loans.</p><p>The immediate goal is to gain a sense of how to create a <a href="https://en.wikipedia.org/wiki/Financial_technology">financial technology</a> (<em>fintech</em>) product from start to finish. The larger goal is to make loan applications a transparent process.</p><p>Before introducing the characters in this system, please bear in mind that behind the titles everyone involved is human. This means that not everything can be quantified and, in many cases, the ultimate decision might be based on human judgement.</p><h4>Creditors</h4><p>Creditors are those with money to lend, and must be careful not to squander their assets on dead-end endeavors. Traditionally, lenders would rely on the 5 Cs of credit:</p><ul><li>Character</li><li>Capacity</li><li>Capital</li><li>Conditions</li><li>Collateral</li></ul><h4>Applicants</h4><p>A loan applicant needs to prove their credit worthiness. The information needed can vary from lender to lender, and navigating the application process can be perplexing.</p><p>The need for both sides of the marketplace to understand the risks and rewards is therefore clearly apparent. In this post we will use Lending Club data to create <strong>LoanHelp</strong>: a simple website capable of helping both creditors and borrowers find out if that loan is worth it or not.</p><p><em>Feel free to check out the GitHub repo for the detailed analysis: </em><a href="https://github.com/NadimKawwa/loan-help"><em>https://github.com/NadimKawwa/loan-help</em></a></p><h3>Data Acquisition</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RXAQdBMXEQjzf1VG" /><figcaption>Photo by <a href="https://unsplash.com/@thepaintedsquare?utm_source=medium&amp;utm_medium=referral">Jessica Lewis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For this project we are going to use data from <a href="https://www.lendingclub.com/">Lending Club</a>, an American peer-to-peer lending company, collected from 2007 to 2018. The data is available for free download on <a href="https://www.kaggle.com/wendykan/lending-club-loan-data">Kaggle</a>. The files contain information on all loans issued, such as current loan status and the latest payment information.</p><p>The data also contains features such as credit score, number of financial inquiries, zip code, and collections. Overall we have around 890,000 observations and 75 features. To reiterate, our goal is to predict whether or not an individual will pay back their loan or default.</p><p>Prior to conducting any steps, we sort by issue date and make the last 10% of observations the test set. For better visualization consider Figure 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*hNEHt0PHaUT6J_2YGpfeRg.png" /><figcaption>Figure 1: Loan Issued Per Year</figcaption></figure><p>This method entails that the testing data will not be seen in the exploration phase, adding another safeguard against leakage. This is common practice as we have no means to know for sure what the future is.</p><blockquote>Our goal is to predict whether or not an individual will pay back their loan or default</blockquote><h3>Data Exploration &amp; Feature Engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pucu33Q1gH-VlRos" /><figcaption>Photo by <a href="https://unsplash.com/@ryanquintal?utm_source=medium&amp;utm_medium=referral">Ryan Quintal</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Future Leak</h4><p>Some information in the data occurs in the future after the loan has been issued, such as installment: the monthly payment owed by the borrower if the loan originates. This information is indicative of the future and constitutes a data leak and should be discarded. Overall, in this step we discarded about <strong>50</strong> features from the analysis.</p><h4>Missing Observations</h4><p>The dataset contains a lot of missing inputs, and it is tempting for data scientists to discard a feature if more than half are missing. However, it is advisable to take one’s time and give each feature its due diligence. There is joy in uncovering patterns amidst chaos, it’s what data scientists do.</p><p>Figure 2 below shows the distribution of missing observations as a percentage of each feature. We make the judgement to drop all the features that have more than 50% missing values. For those with less than 10% missing we fill in the mode of that feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*KvL6Bae1E0DCGyulORB7GQ.png" /><figcaption>Figure 2: Mean Missing Values Per Column</figcaption></figure><h4>Categorical Data</h4><p>Some features are categorical data such as job titles. For the latter and similar features, we engineer our own features that attempt to assign jobs to relevant categories. For example, titles such as Nurse and Doctor tick the mark as ‘<strong>health</strong>’ occupations. In addition, we created a binary feature for ‘<strong>executives</strong>’ who have titles such as president, CEO, and regional manager.</p><h4>Custom Feature Scaling by Geography</h4><p>For many machine learning algorithms, a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">standard scale</a>r instance can boost performance. Before diving in, consider the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*Od-laHtbPygd0fX_" /><figcaption><a href="https://www.womansday.com/life/a51100/the-history-and-meaning-of-zip-codes/">Source: womansday.com</a></figcaption></figure><p>It makes more economic sense to scale observations by <a href="https://en.wikipedia.org/wiki/Metropolitan_statistical_area">metropolitan statistical area</a> (MSA) than the whole nation. If the first digit of a zip code indicates a state, the second two digits denote the central post office facility for that area.</p><p>We shall therefore keep track of statistics such as the mean and standard deviation of samples grouped by the first 2 digits that we shall denote as the ‘macro’ statistic. The reasoning behind this is that $20,000 has a different value in Austin, TX than say, San Francisco, CA (the author wishes he paid less rent).</p><h4>User Profiles</h4><p>So who are the debtors of Lending Club? The company assigns each a letter sub grade (more on that later), such that A1 is the best possible grade and G5 is the worst. The distribution appears to be skewed to the right, with most users having OK to good grades.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*SOiajKpdE1KYcyLLJGsaTA.png" /></figure><p>One interesting result is the plot below comparing annual income of paid and default loans. Note that the distributions seem identical, whereas those who tend to default have on average a slightly lower income.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*j8G2BGpnHhzh0U3RuAeZEg.png" /></figure><p>Perhaps then the two plots are of greater interest. We see that a higher interest rate (int_rate) correlates with a higher likelihood of default. In parallel, a lower sub grade is also indicative of a possible default. We have presented these two plots together since applicants with a lower sub grade are deemed risky and are subject to a higher interest rate. This is therefore an example of <a href="https://en.wikipedia.org/wiki/Feature_selection">feature correlation</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*kd0vob7WwZ838Q_HuOTcYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*27SBUFtuKJCaPsbzRrlvSA.png" /></figure><h3>Modeling &amp; Performance Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4rMLFBjyHRqwcCkVOO7mig.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@ryanquintal?utm_source=medium&amp;utm_medium=referral">Ryan Quintal</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Picking a Model</h4><p>With the data cleaned, we opted to try several supervised learning algorithms in order to pick a winner and develop it further.</p><h4>Performance Metrics</h4><p>Given the imbalanced nature of the dataset, relying on accuracy alone is not enough. Metrics such as <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">Receiver Operating Characteristic </a>(ROC) and the <a href="https://en.wikipedia.org/wiki/F1_score">f1 score</a> are useful in the case of imbalanced datasets. Figure 3 below shows the ROC curve for different models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*efuf69TUpFwkIh63TrFuHQ.png" /><figcaption>Figure 3: Preliminary ROC Curve</figcaption></figure><p>Note that each classifier was instantiated with the default parameters, therefore results might vary depending on the parameters. In a more advanced study, one would compare the best of each model against the best of other models. For the sake of limiting the scope of the project, we decided to go with XGBoost given its overall superior performance. Overall XGBoost was able to achieve an AUC of 0.68.</p><h4>Optimizing Results</h4><p>As with all machine learning algorithms, cross validation and grid search are useful tools to get the most out of performance. Hyperparameter tuning can be a computationally expensive process, it is therefore preferably done with a <a href="https://colab.research.google.com/">google colab</a> notebook. The model hyperparamteres were tuned, however, only for a small number of trials.</p><h3>Interpretation of Output</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HfLfRy9aVBIZ2SxMGrpuLQ.jpeg" /><figcaption>Source: <a href="https://en.wikipedia.org/wiki/United_States_fifty-dollar_bill">Wikipedia</a></figcaption></figure><p>The Federal Trade Commission (FTC) is a US agency in charge of consumer protection and regulates business practices, among other things. Depending on where you live, there is probably a government agency with similar tasks.</p><p>The FTC may require an institution to justify why certain loan requests were denied or face legal ramifications. Therefore, model explainability is paramount for legal compliance. Furthermore, if any feature relates to personal information like gender or ethnicity: drop them. It’s not just unethical, it’s also against the law.</p><p>From a product standpoint, users on the website might want to understand why their application did not go through. Could it be their age? Credit score? Were there any elements in their favor? Frustrated users tend to abandon the platform altogether. Opaque machine learning models are a guaranteed way to lose business. Explainability is not just the law, it is also key to growth.</p><p>Hence this section deals with understanding what are the underlying features at play and how they sway the model’s decision.</p><blockquote>Opaque machine learning models are a guaranteed path to poor decisions</blockquote><p>We make use of SHAP values in order to gain a case by case explanation of model outputs. Note that SHAP feature importance is quite distinct from feature permutation importance. Indeed, permutation is based on the decrease in in model performance whereas SHAP is based on the magnitude of feature attributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*6Oou-Ynz0JklsoSIAcaaIw.png" /><figcaption>Figure 4: SHAP Summary Plot</figcaption></figure><p>Figure 4 is a density scatter plot of SHAP values for each feature to identify how much impact each feature has on the model output for individuals in the training dataset.</p><p>Figure 4 limits the output to the top 30 features sorted by the sum of the SHAP value magnitudes across all samples. The top feature is the “subgrade” (a Lending Club assigned grade), such that the best grade is A1 and the worst is G5. Intuitively, it makes sense to have the best rated applicants the most likely to pay off their loans.</p><p>The next feature is “term” and refers to the number of payments on the loan, such that values are in months and can be either 36 or 60. The third feature “dti” is “<em>a ratio calculated using the borrower’s total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower’s self-reported monthly income”</em>.</p><p>We move our attention to home ownership, which can be one of owns, mortgage, rents, any, other, or none. In the feature engineering portion of the project, we decided to replace these observations with a numerical value such that owning a home has the highest value. We bring our attention to this feature because for a subset of users, being a homeowner can be an indicator of a pending default. It is worth noting here that feature interaction might play a role in this assessment.</p><h3>Putting it All Together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MzwUNrc-IDV5Ga2u" /><figcaption>Photo by <a href="https://unsplash.com/@precondo?utm_source=medium&amp;utm_medium=referral">Precondo CA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>With our model trained and results up to satisfaction, we can wrap it up into a script where users can input information about themselves and receive a credit assessment.</p><blockquote>The larger goal is to provide a case-by-case explanation, to make loan applications a transparent process</blockquote><p>We must take into account what we can reasonably ask of the users. For example, what does a sub grade of C1 mean? How does a user know where they stand? Therefore, we refer to <a href="https://www.sec.gov/Archives/edgar/data/1409970/000089161808000318/f41480orsv1.htm#111">Lending Club’s SEC filing </a>which provides a table of sub grade versus average FICO score (see page 49). Below is a plot of this relationship between sub grade and FICO:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*zdq8JgZtMELENood1xTjMg.png" /><figcaption>Figure 5</figcaption></figure><p>Note that this chart is not exactly as robust as we want it to be. Notice that there are kinks in the plot that defy logic: F1 has a better FICO score than E5. We would expect an almost total positive linear correlation between the FICO score and sub grade. There are other considerations for how to calculate the sub grade, perhaps hidden in other parts of the SEC filing. However, for the context of this study we shall consider this information sufficient.</p><p>We must also take into account user fatigue for this portion and the scope of our application. In terms of app development, can we ask the user to input 30+ pieces of information about themselves for a preliminary evaluation? For practical considerations, we will cap the required inputs to key items such as:</p><ul><li>FICO score</li><li>Location</li><li>Debt to Income</li><li>Loan Amount</li><li>Annual Income</li></ul><h4>Demonstration</h4><p>We make use of python’s <a href="https://flask.palletsprojects.com/en/1.1.x/">Flask</a> framework to create a simple website. We then use <a href="https://www.heroku.com/">heroku</a>, a cloud platform as a service supporting several programming languages, to deploy it online:</p><p><a href="https://loan-help.herokuapp.com/">Lend Help: Preliminary Loan Approval Help</a></p><p>Below is a demonstration of how to use the website. We begin by filling out information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*1NzIFLjy0GaBv1hM3zxwhw.png" /></figure><p>Hitting the submit button results in a decision and information about the loan:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*b34KeqVu9oXDuvGiIDOsSQ.png" /></figure><h4>Considerations When Developing A Web App</h4><p>The most important lesson for developing an application on Heroku or anywhere else: Always specify the python dependencies with <a href="https://devcenter.heroku.com/articles/python-pip#best-practices">versions explicitly specified</a>.</p><p>In addition, consider the amount of traffic expected and how to reduce computation time. For example, loading the pickled model as soon as the page is opened saves us the effort of having to load it every time we make a prediction. On the other hand, the website is hosted on a free domain that sleeps after 30 minutes of inactivity.</p><p>Finally, consider what input types and values we expect and place hard restrictions where needed. W3schools provides an <a href="https://www.w3schools.com/html/html_form_input_types.asp">easy tutorial </a>for those looking to get started.</p><h3>Defaults and Insurance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_vTP1h5SzyT7EkS8" /><figcaption>Photo by <a href="https://unsplash.com/@jpvalery?utm_source=medium&amp;utm_medium=referral">Jp Valery</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So far we have been able to explain the model’s output in such a way that is digestible for creditors and loan applicants. We can further extend our findings into a practical method to assess risk and increase revenue for creditors. In this section we will briefly show how data science can be applied to loan insurance.</p><p>Some creditors might want their assets insured in case of default. How do we calculate the expected gain? Let’s assume a simple case where an applicant is given a $1000 loan.</p><p>At the end of each month, the debtor is expected to pay the money back with interest. If there is a 5% chance of default at any month then the expected profit for the creditor is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*DKiNqu1Ws_Wp5Oj-gUw3Gg.png" /></figure><p>Where <strong>n_paid</strong> is the number of times the end of month fee was paid, and <strong>n_unpaid</strong> the number of missed payments. If all billing cycles are paid off, the credit will get their money back plus interest. If at any point the debtor defaults on the loan, any future payments are considered lost.</p><p>How does this apply on a larger scale? Insurers will have a mix of reliable and less than stellar debtors. By pooling these portfolios together, insurers can offer lower rates since the loss from defaults is offset by premiums paid in full by worthy debtors. Data science plays a key role here since it produces a probability of default for each loan.</p><h4>Scoring and Confidence</h4><p>Typically, we want to present our best performance metrics and call it a day. However, will the deployed model consistently return the same results? How can we guarantee a minimum performance to stakeholders?</p><p>Using the .632 Bootstrap method by Bradley Efron and Robert Tibshirani, we are able to construct confidence intervals. We found out that 95% of the time, the AUC is between <strong>(0.64, 0.67)</strong>.</p><h3>Scaling the Product</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5LV2Hhq3LZIo32VU" /><figcaption>Photo by <a href="https://unsplash.com/@timbatec?utm_source=medium&amp;utm_medium=referral">Pepi Stojanovski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The machine learning model would eventually reside on dedicated a web server and be called upon by creditors and debtors who use the service. Setting aside the time needed to load the webpage and access the data, how much time does it take to make a decision?</p><p>We performed this exercise on the test data and found that on average a prediction is made in <strong>203 microseconds</strong> with a standard deviation of 62<strong> </strong>microseconds. Performance can probably be optimized, however the time is quite reasonable.</p><h3>Conclusion and Looking Ahead</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TmXoT95xRb7ZCpNp" /><figcaption>Photo by <a href="https://unsplash.com/@blankerwahnsinn?utm_source=medium&amp;utm_medium=referral">Fabian Blank</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Using consumer data, we were able to help both sides of the marketplace assess the outcome of a loan and make wiser decisions. The inputs and results are given in plain English and do not require expert knowledge.</p><p>Despite our best efforts, we could not achieve a model that is almost always right. This is in part due to the fuzzy nature of the problem. In addition, some critical features are not present in the data such as age, education level, and micro/macro economic factors.</p><p>As with any machine learning model, it is crucial to regularly update our training data. Over time consumer trends change, new products become available, the ground truth shifts, and as a result our metrics start taking a hit.</p><p>How often we retrain the model and which data to use is beyond the scope of this article, but a necessary consideration prior to production deployment.</p><p>Hope you enjoyed reading this article!</p><p><em>Got questions? Comments? Feedback? Or do you just want to discuss things further? Feel free to reach out and I’ll do my best to answer you promptly!</em></p><h3>References</h3><ul><li><a href="https://www.nerdwallet.com/blog/5-cs-credit/">Master the 5 C&#39;s of Credit - NerdWallet</a></li><li><a href="https://www.kaggle.com/wendykan/lending-club-loan-data">Lending Club Loan Data</a></li><li><a href="https://www.consumer.ftc.gov/topics/credit-and-loans">Credit and Loans</a></li><li><a href="https://christophm.github.io/interpretable-ml-book/shap.html">9.6 SHAP (SHapley Additive exPlanations) | Interpretable Machine Learning</a></li><li><a href="https://www.investopedia.com/articles/pf/08/loan-protection-insurance.asp">Is Loan Protection Insurance Right For You?</a></li><li><a href="https://www.inc.com/encyclopedia/insurance-pooling.html">Insurance Pooling</a></li><li><a href="https://en.wikipedia.org/wiki/Expected_value">Expected value - Wikipedia</a></li></ul><p><a href="https://www.cs.nmsu.edu/~hcao/teaching/cs488508/readings/EfronTibshirani_JASA_1997.pdf">Improvements on Cross-Validation: The .632+ Bootstrap Method, Bradley Efron and Robert Tibshirani</a></p><p><a href="https://www.doctorofcredit.com/use-lending-club-find-fico-score-range-free/">Find Out Your FICO Score For FREE By Using Lending Club</a></p><p><a href="https://www.sec.gov/Archives/edgar/data/1409970/000089161808000318/f41480orsv1.htm">https://www.sec.gov/Archives/edgar/data/1409970/000089161808000318/f41480orsv1.htm</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0eda750bed9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-issue-smarter-loans-d0eda750bed9">How to Issue Smarter Loans</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Perform Linear Regression With Confidence]]></title>
            <link>https://medium.com/data-science/how-to-perform-linear-regression-with-confidence-5ed8fc0bb9fe?source=rss-85e7e8816e35------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ed8fc0bb9fe</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Nadim Kawwa]]></dc:creator>
            <pubDate>Tue, 24 Mar 2020 16:38:24 GMT</pubDate>
            <atom:updated>2020-03-24T16:38:24.880Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why It Works &amp; Where It Fails</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ptWVVat3tagJC5XJ" /><figcaption>Photo by <a href="https://unsplash.com/@grakozy?utm_source=medium&amp;utm_medium=referral">Greg Rakozy</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Linear regression is perhaps one of the most widely used approaches to model relationships. Its main draw comes from the fact that it is easy to explain to non-technical readers and the assumption that relationships are linear.</p><p>In this post we are going to explain the theory and assumptions for linear regression and implement code that yields similar results to <a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.linregress.html">scipy.stats.linregress</a>.</p><p>The code is written in Python 3.x and the whole workflow can be found here: <a href="https://github.com/NadimKawwa/Statistics/blob/master/linear_regression.ipynb"><em>https://github.com/NadimKawwa/Statistics/blob/master/linear_regression.ipynb</em></a></p><h3>Assumptions</h3><p>Below are assumptions about the <a href="https://methods.sagepub.com/reference/encyc-of-research-design/n329.xml">predictor variables</a>, the <a href="https://www.quora.com/What-is-a-response-variable-in-statistics">response variables</a>, and the relationship between them:</p><ul><li>Weak <a href="https://en.wikipedia.org/wiki/Exogeny">Exogeneity</a>: We assume that the predictor variables are error free, for example that there are no field measurement errors.</li><li>Linearity: The mean of the response variable is a linear combination of the parameters and the predictor variables.</li><li><a href="https://en.wikipedia.org/wiki/Homoscedasticity">Homocedasticity</a>: Different values of the response variable have the same variance in their errors (homogeneity of variance).</li><li>Independence of Errors: Errors of the response variables are uncorrelated with each other.</li><li>Lack of Perfect Multicolinearity: For least square estimation methods, the design matrix X must have <a href="https://en.wikipedia.org/wiki/Rank_(linear_algebra)#Main_definitions">full rank</a>, otherwise it could mean that one predictor variable is given twice.</li></ul><h3>Method of Least Squares</h3><h4>Theory</h4><p>In practice, there are many problems where a set of paired data gives the indication that regression is linear, where we do not know the <a href="https://en.wikipedia.org/wiki/Joint_probability_distribution">joint distribution</a> of the random variables under consideration.</p><p>The <a href="https://en.wikipedia.org/wiki/Least_squares">method of least squares</a> is a curve fitting method to approximate linear relationships.</p><p>Consider the table below that shows the test scores versus the hours studied for 10 random students:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*hL0jJaDii8GMkVjn_d0qDA.png" /></figure><p>In the image below we show a scatter of hours studied versus test scores:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*Uq7iGRP9JPfeNyLkxSMnow.png" /></figure><p>We get the impression that a straight line provides a reasonably good fit such that the estimated regression line is given as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*63Zi8V1KhkmWS7F_B8vOUQ.png" /></figure><p>Say we have fitted an arbitrary line that estimates the linear relationship rather well. How do we determine the goodness of fit? Let <strong><em>ei</em></strong> denote the vertical deviation from a point to the estimated regression line. We therefore want to minimize the quantity q with respect to <strong><em>α</em></strong> and <strong><em>β</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*h14xk2IrNJxAUN1or1SDKw.png" /></figure><p>Without going into the details of differentiating q partially with respect to α and β, we will skip directly to the solution and introduce the following notations to simplify future calculations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*K0gWOlr93B6c2ZMLNiwWTg.png" /></figure><p>Given sample data (xi, yi), the coefficients of least square α and β are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*IcjGvVfeIqgWJbxV3CtMUg.png" /></figure><h4>Implementation</h4><p>In the code below we implement this simple linear regression. We first define a helper function getPairStats to calculcate Sxx, Syy, and Sxy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d4973587f25120a3ea74881ed614205/href">https://medium.com/media/7d4973587f25120a3ea74881ed614205/href</a></iframe><p>Implementing the linear regression becomes straightforward now:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d132201e082f8860bfb2fe261203f967/href">https://medium.com/media/d132201e082f8860bfb2fe261203f967/href</a></iframe><p>In order to gain more insight into what we are doing, consider the plot below where we show the result of fitting a line and the vertical deviations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*SoxlmCtjjcnKl46aMwBWzw.png" /></figure><h3>Hypothesis Testing</h3><h4>Theory</h4><p>In normal regression analysis, it is assumed that for each fixed Xi the condition density of the corresponding random variable Yi is the normal density:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*W2G3UG_LBdq-uqTEsUJCFQ.png" /></figure><p>Such that α, β, and σ are the same for each i. Given a random sample of paired data, normal regression analysis concerns itself with the estimation of σ and the regression coefﬁcients α and β, with tests of hypotheses concerning these parameters.</p><p>The maximum likelihood estimate of σ is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*yeh8cayUicoKAFoRy6sBmQ.png" /></figure><p>More often than not, we are interested in testing the slope coefficient β. We note that β hat has a normal distribution and under the assumption of normal regression analysis the value of a random variable having the t distribution with (n−2) degrees of freedom is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*CoUb68TDWyTrBnfvhb9FtA.png" /></figure><p>In order to build a (1-α) confidence interval for the parameter β:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*ULf7GW41eROG9OAGKk5sNQ.png" /></figure><h4>Implementation</h4><p>Let us now test the following hypothesis on the hours studied versus score:</p><ul><li>H0: β=3</li><li>H1: β&gt;3</li><li>Set the significance level α=0.01</li></ul><p>The function below builds upon the pair regression function from the previous section and returns False if we fail to reject H0, True otherwise.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf3da2290c759b0db6cf9a3e624e1cfb/href">https://medium.com/media/cf3da2290c759b0db6cf9a3e624e1cfb/href</a></iframe><p>With reference to the same data, we can construct a 95% confidence interval for β:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05940c22c2e0c4fe836b0e488f7eea3b/href">https://medium.com/media/05940c22c2e0c4fe836b0e488f7eea3b/href</a></iframe><p>How would our regression look with these slopes, keeping the intercept constant? The plot below provides a visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*Xu7oLQhuJAQjgzN40aoJnw.png" /></figure><h3>Normal Correlation Analysis</h3><h4>Theory</h4><p>The sample correlation coefficient, denoted as r, measures the strength of the linear relationships between two sets of data X and Y. When r=0 then the random variables are uncorrelated. the coefficient is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*6nFwq8xiKDwf4UMrCgzkVw.png" /></figure><p>With the helper function getPairStats, calculating r is very easy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad79e116c28ad22cf0e6dc6df2356c25/href">https://medium.com/media/ad79e116c28ad22cf0e6dc6df2356c25/href</a></iframe><h3>Pitfalls</h3><p>The limitations of linear regression are evident in the assumptions. In addition, one must recall that correlation does not imply necessary causation.</p><p><a href="https://en.wikipedia.org/wiki/Anscombe%27s_quartet">Anscombe’s quartet</a> is 11 data points (x,y) that are designed to have nearly identical means, standard deviations, and correlations. The results are identical regression lines that demonstrate the need to plot the data before graphing and the impact of outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*wlwgf3JMoFB4wXau.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Anscombe%27s_quartet">Anscombe&#39;s quartet</a></figcaption></figure><p>Another consideration is seasonality, especially for time series plots. It might not make sense to have one slope coefficient for all the time spans.</p><h3>Multiple Linear Regression</h3><h4>Theory</h4><p>In real applications, we will face many situations where multiple variables are used to approximate a target outcome. In this section we will generalize simple linear regression into a multi linear model.</p><p>We can use matrix notations in order to solve multiple linear regression. Let X be an (n, k+1) matrix consisting of the given values with the first column appended to accommodate constant terms. Y is an (n,1) matrix, i.e. a column vector, consisting of the observed values of Y. And B is a (k+1, 1) matrix consisting of the least squares estimates of the regression coefficients.</p><p>In matrix format X, Y, and B are written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*JGxDWo5kXA_y5M_Qxlv76w.png" /></figure><p>The least squares estimate of the multiple regression coefficients are given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*430PET4jPcFsVEpLVFSApw.png" /></figure><h4>Implementation</h4><p>Consider an attempt to predict the price of a home using the number of bedrooms and number of baths:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*8i3jsiuuMAGGTVEqg79kXA.png" /></figure><p>To save on computation time later, we define a helper function calculate (X.T X)-¹. This term will prove to be important in the next section:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecdb0e2590eef69d64bcccd554c947bd/href">https://medium.com/media/ecdb0e2590eef69d64bcccd554c947bd/href</a></iframe><p>The equations from above are interpreted in the compact function below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b784451fcda136a51dc50121086cc1d/href">https://medium.com/media/2b784451fcda136a51dc50121086cc1d/href</a></iframe><h3>Multiple Hypothesis Testing</h3><p>With multiple coefficients comes the need to test each of these coefficients against a null hypothesis. In the example above, the coefficient B1=15,314 implies that each bedroom adds about ~$15,000 to the asking price.</p><p>The t statistic with (n-k-1) degrees of freedom is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*mUOQ3Io_7R9rKDc3VjHhJQ.png" /></figure><p>Such that c_ii is the ith row and ith column of the matrix (X.T X)-¹ and:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*Nnh7Z6eFz-qEb71_OwoNJQ.png" /></figure><h4>Implementation</h4><p>Suppose we wish to test the following hypothesis at :</p><ul><li>H0: B1=9500</li><li>H1: B1&gt;9500</li><li>Set the significance level α=0.05</li></ul><p>Implementing it in a function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5b5bb52902f37096a2feee723b3b5ed/href">https://medium.com/media/b5b5bb52902f37096a2feee723b3b5ed/href</a></iframe><p>We conclude that on average each bedroom adds more than $9500 to the asking price of a house.</p><h3>Conclusion</h3><p>Linear regression is a useful tool to approximate relationships, and is widely used in all scientific domains. Its simplicity and ease of explanation make it a preferred choice for conveying results to a technical audience and laypeople.</p><p>There are of course more regression methods such as KNN regression and Random Forest regression.</p><h3>References</h3><p>John E. Freund, <em>Mathematical Statistics with Applications</em>, 8th Edition</p><ul><li><a href="https://en.wikipedia.org/wiki/Simple_linear_regression">Simple linear regression - Wikipedia</a></li><li><a href="https://en.wikipedia.org/wiki/Linear_regression">Linear regression - Wikipedia</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ed8fc0bb9fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-perform-linear-regression-with-confidence-5ed8fc0bb9fe">How to Perform Linear Regression With Confidence</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>