<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naman Doshi on Medium]]></title>
        <description><![CDATA[Stories by Naman Doshi on Medium]]></description>
        <link>https://medium.com/@namandoshi?source=rss-a42d268507e9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JVs_4nvdx_h0Ep29lzx_Fw.png</url>
            <title>Stories by Naman Doshi on Medium</title>
            <link>https://medium.com/@namandoshi?source=rss-a42d268507e9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:52:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@namandoshi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solving real-world problem using data science]]></title>
            <link>https://medium.com/data-science/solving-real-world-problem-using-data-science-e2236159a1a9?source=rss-a42d268507e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2236159a1a9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[problem-solving]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Naman Doshi]]></dc:creator>
            <pubDate>Thu, 18 Oct 2018 17:46:31 GMT</pubDate>
            <atom:updated>2018-10-18T17:46:31.670Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*NnHVDbXKAubGx9L7.jpg" /><figcaption>Source: Betsol</figcaption></figure><p>The world of data science is evolving every day. Every professional in this field needs to be updated and constantly learning, or risk being left behind. You must have an appetite to solve problems. So I decided to study and solve a real-world problem which most of us have faced in our professional careers. The technical round in an interview!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*QEDj--ZI8lqNDqFTJAAekw.gif" /></figure><p>How many times have you gone through a technical interview where you feel you’re acing it, and then a question comes that leaves you stumped? And from there the entire interview goes downhill because now you have lost confidence and the recruiter has lost interest.</p><p>But is it fair to judge the technical capabilities of a candidate based entirely on a 3-hour interview? This is a loss at both the ends because now the company has lost a potential candidate and the candidate has lost an opportunity.</p><p>If only there was a way through which the recruiter can get the gist about the technical capabilities of the candidate outside the interview hall. <br>A scoring system of sorts — that would give an ideal score to gauge the technical knowledge of the candidate, and thereby help the recruiter to make an informed, unbiased decision. Sounds like a dream scenario, right?</p><p>So I decided to start a project called “Scorey” that aims to crack this challenge.</p><p><strong>Scorey helps in scraping, aggregating, and assessing the technical ability of a candidate based on publicly available sources.</strong></p><h3><strong>Setting the Problem Statement</strong></h3><p>The current interview scenario is biased towards “candidate’s performance during the 3-hour interview” and doesn’t take other factors into account, such as the candidate’s competitive coding abilities, contribution towards the developer community, and so on.</p><h3>The Approach We’ll Take</h3><p>Scorey tries to solve this problem by aggregating publicly available data from various websites, such as:</p><ul><li>Github</li><li>StackOverflow</li><li>CodeChef</li><li>Codebuddy</li><li>Codeforces</li><li>Hackerearth</li><li>SPOJ</li><li>GitAwards</li></ul><p>Once the data is collected, the algorithm then defines a comprehensive scoring system that grades the candidates technical capabilities based on the following factors:</p><ul><li>Ranking</li><li>Number of Problems Solved</li><li>Activity</li><li>Reputation</li><li>Contribution</li><li>Followers</li></ul><p>The candidate is then assigned a scored out of 100. This helps the interviewer get a full view of a candidate’s abilities and hence make an unbiased, informed decision.</p><h3>Setting up the Project</h3><p>For the entire scope of this project, we are going to use Python, a Jupyter notebook &amp; scraping libraries. So if you’re someone who likes Notebooks, then this section is for you. If not, feel free to skip this and move on to the next section.</p><pre><strong>def color_negative_red(val):<br>    <em>&quot;&quot;&quot;</em><br><em>    Takes a scalar and returns a string with</em><br><em>    the css property `&#39;color: red&#39;` for negative</em><br><em>    strings, black otherwise.</em><br><em>    &quot;&quot;&quot;</em><br>    color = &#39;red&#39; if val &gt; 0 else &#39;black&#39;<br>    return &#39;color: %s&#39; % color</strong></pre><pre><strong><em>( Set CSS properties for th elements in dataframe)</em><br>th_props = [<br>  (&#39;font-size&#39;, &#39;15px&#39;),<br>  (&#39;text-align&#39;, &#39;center&#39;),<br>  (&#39;font-weight&#39;, &#39;bold&#39;),<br>  (&#39;color&#39;, &#39;#00c936&#39;),<br>  (&#39;background-color&#39;, &#39;##f7f7f7&#39;)<br>  ]<br><br><em># Set CSS properties for td elements in dataframe</em><br>td_props = [<br>  (&#39;font-size&#39;, &#39;15px&#39;),<br>    (&#39;font-weight&#39;, &#39;bold&#39;)<br>  ]<br><br><em># Set table styles</em><br>styles = [<br>  dict(selector=&quot;th&quot;, props=th_props),<br>  dict(selector=&quot;td&quot;, props=td_props)<br>  ]</strong></pre><p>This will make your dataframe output look neat, tidy, and really good!</p><p>Now that we have a gist of what we are aiming to solve and how we are going to go about it, let’s code!</p><h3>Step 1: Scraping Personal Website</h3><p>We need to aggregate the entire “<em>coding presence of a person on the internet</em>”. But where do we start? Duh! His/Her personal website. <em>This is of course assuming we have access and permission to the candidate’s personal website.</em> We can parse all the necessary links from there.</p><pre><strong>from</strong> <strong>bs4</strong> <strong>import</strong> BeautifulSoup<br><strong>import</strong> <strong>urllib</strong><br><strong>import</strong> <strong>urllib.parse</strong><br><strong>import</strong> <strong>urllib.request</strong><br><strong>from</strong> <strong>urllib.request</strong> <strong>import</strong> urlopen<br><br>url = input(&#39;Enter your personal website - &#39;)<br><br>html = urlopen(url).read()<br>soup = BeautifulSoup(html, &quot;lxml&quot;)<br><br>tags = soup(&#39;a&#39;)<br><br><strong>for</strong> tag <strong>in</strong> tags:<br>    print (tag.get(&#39;href&#39;,<strong>None</strong>))</pre><p>When we run this piece of code, we get the below output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*3zsFNp3gxDC1RGCWWuLLGw.png" /></figure><p>Here we are using <em>BeautifulSoup </em>which is a popular scraping library. Using this block of code, we have direct links to a candidate’s online profile.</p><p>Now where will you begin if you had to assess a coder at a much more granular level?</p><ol><li><strong>Github</strong></li></ol><p>So first, let us use Github API to get all the info we need of a particular user.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb5f6e17b53e50899c89f83ff372c14c/href">https://medium.com/media/eb5f6e17b53e50899c89f83ff372c14c/href</a></iframe><p>For our use case, we only need email, number of repositories, followers, hireable (true or false), current company and last recorded activity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*_q1hGsjB6JsTc1mrEVppHA.png" /></figure><p>2. <strong>StackOverflow</strong></p><p>Ah. Devs might not believe in God but StackOverflow is definitely a temple for them. As you may already know, it is very difficult to get a reputation on StackOverflow. For this step, you can use StackExchange’s API — it gives you user data such as reputation, no. of answers, etc.</p><p>We’ll then add these new attributes to our existing dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*PM2T2Bxfi1_RfCfaJdQE9A.png" /></figure><p>Now we are going to target and scrape global competitive programming platforms such as CodeChef, SPOJ, Codebuddy, Hackerearth, CodeForces &amp; GitAwards (for a deeper insight into their projects).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ae483758087a5e461b915e4c142ae96/href">https://medium.com/media/1ae483758087a5e461b915e4c142ae96/href</a></iframe><p>All this scraping gave us a LOT of info as you can see. Code is pretty self explanatory. I’ve also documented using comments so that its easy to understand.</p><p>Without going into the nitty-gritty of the code, I’d like to focus on the process. But you can give me a shout-out if you face any trouble executing the code. :) Now that we have all the data in hand, we will move on to creating a scoring algorithm.</p><h3>Step 2: Scoring System</h3><p>The next part is to score the candidates on the following parameters:</p><ul><li>Rank (25 points)</li><li>Number of problems solved (25 points)</li><li>Reputation (25 points)</li><li>Followers (15 points)</li><li>Activity (5 points)</li><li>Contributions (5 points)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74b27b9f5f68fa7fb7dbe33f219fb8e4/href">https://medium.com/media/74b27b9f5f68fa7fb7dbe33f219fb8e4/href</a></iframe><p>So if you go through this piece of code, you’d understand how we can create a scoring system. Though its pretty basic at this point, we can use machine learning to create a robust dynamic scoring system.</p><p><strong>Final Score</strong></p><p>Based on the point system we saw above, the algorithm will now assign a final score to the candidate’s technical capabilities.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64c36ac9bdfd1f2c096c637197c574d7/href">https://medium.com/media/64c36ac9bdfd1f2c096c637197c574d7/href</a></iframe><p>So the user <em>poke19962008</em> has a <strong>score of 64 out of 100</strong>! Now this will give the recruiters an idea of the technical abilities of the candidate <em>outside the interview room.</em></p><h3>Step 3: Predictive Modeling</h3><p>When you are trying to solve a real world problem and “<em>productivize</em>” the solution, its important to consider the requirements of the end user.<br>In this case, its the recruiter. <br><em>How can we use power of machine learning to add value to the recruiter?</em></p><p>Upon brainstorming, I found the following use cases — <br>1. Model that predicts whether or not the management will be satisfied by candidate’s skill set<br>2. Model that predicts the probability of a candidate’s churn post hiring<br>3. Using genetic algorithm to link assign the candidate to respective team</p><p>Let’s try to code the 1st use case — Predicting company’s satisfaction<br>Assuming that the recruiter has been using Scorey to screen candidates for some time and now has a database of 100 candidates.<br>Post recruitment, based on the candidate’s performance, the recruiter updates the database with a new binary attribute “Satisfaction” with values of either 0 or 1.<br>Let’s create a dummy database for now and try to create a model using <em>Scikit-Learn, Pandas, Numpy</em> and build a predictive model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1d5d3fa6d2bc6f0d7fc238b0d44e950/href">https://medium.com/media/d1d5d3fa6d2bc6f0d7fc238b0d44e950/href</a></iframe><ol><li>Import data &amp; libraries</li><li>Clean the data — remove duplicates and null values</li><li>Using label encoder to deal with categorical data</li><li>Split the dataset into train &amp; test</li><li>Using kNN classifier to predict</li><li>Check Accuracy</li></ol><p>Using these steps, you will get a niche model that will be able to predict whether the candidate will fit into the company based on underlying trends.<br>For eg — Candidates who have higher reputation and are contributing to Open source are more likely to retain for a longer period of time.</p><h3>Step 4: Dashboarding</h3><p>I went ahead and made a dashboard. This its still a work-in-progress and I’ll be happy to share some of the screenshots of the interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*GzwtDv_zJBDrzmGuqknwWA.gif" /></figure><p>There you have it. Your very own end-to-end product.<br>To summarize — <br>1. We identified a problem<br>2. Methodical thinking on how we can solve it<br>3. Used Web scraping to gather data<br>4. Build an algorithmic scoring system<br>5. Machine learning to build a predictive model<br>5. Dashboard to communicate results</p><p>Tech stack that we used — <em>Python: BeautifulSoup, Urllib, Pandas, Sklearn</em></p><p>So that’s all for this article. We took a real life problem and tried to use data and algorithms to solve it!</p><p>Next time you go for an interview, you can pitch this system to the recruiter. :)</p><h3><strong>So what’s next?</strong></h3><p>Code for the entire project can be found on <a href="https://github.com/warmachine0609/Scorey">Github — here</a></p><ul><li>Integrating Machine learning components for rule generation</li><li>Handling missing data exceptions dynamically</li></ul><p><strong>If you think this project is cool and would like to contribute, you are more than welcome! Let’s build something exciting for the community.</strong></p><p>You can connect with me over<strong> </strong><a href="https://linkedin.com/in/naman-doshi"><strong>LinkedIn </strong></a>or on<strong> </strong><a href="https://twitter.com/gofornaman"><strong>Twitter</strong></a><strong> </strong>to get daily updates on what’s new in data science &amp; machine learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2236159a1a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/solving-real-world-problem-using-data-science-e2236159a1a9">Solving real-world problem using data science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Plan 2018]]></title>
            <link>https://medium.com/data-science-weekly-dsw/data-science-plan-2018-181e838f698e?source=rss-a42d268507e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/181e838f698e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dsw]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Naman Doshi]]></dc:creator>
            <pubDate>Fri, 09 Feb 2018 14:58:43 GMT</pubDate>
            <atom:updated>2018-02-10T04:50:50.027Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>“I want to get into Data Science but don’t know how?” — </em></strong>This is by far the most common expression I hear from people who want to get into this field.</p><p>It is definitely a valid question because there is an ocean of materials, books, MOOCs, classes, tutorials out there. Where to begin? Which one is legit? What’s the starting point? Is it Statistics or Programming or Visualization?</p><p>Honestly, it is a really difficult question to answer since most of the Data Scientist are “Scientists” who are expert in their Business domain and bring in years and years of experience.</p><p>But that doesn’t mean that you can’t do it. It’s just that it will take a hell lot of time if you want to be really good at what you do. You don’t necessarily have to get a doctorate but you should act like one, study like one, practice like one and teach like one.</p><p>A few months back, I came across a subreddit wherein a guy left his graduate school and decided to be a self-learning Data Scientist.</p><p>(Such a bold step right?)</p><p>He said if “I can learn the same things online at a fraction of fees and whole less amount of time, why should I join a University?”</p><p>So I want to share that “Plan” or “Curriculum” that he made. I’m sorry for a being a month late but still, you can catch up. I’ve been following this plan and trust me, it is INTENSE. But I’m also balancing my undergrad studies with this so can’t really complain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*msoWJJrFDYybH7AYuQ7qiw.jpeg" /><figcaption>Source: Reddit</figcaption></figure><p>It starts with Stats, Algebra and Probability and later goes into Python. <br>For R lovers, you can take up John Hopkins course on Coursera. Apart from that I really don’t know anything about R so sorry ;)</p><p>Next it goes into Python for Stats and reading a couple of books for getting a statistical mindset.</p><p>At this point you must be pretty good with Python so might as well do some shorts projects on it. You can follow <a href="https://medium.com/u/b4e93b7cf1b1">ML Review Team</a>’s blog for some kickass ideas.<br>Next, the fun part — Visualization. Followed by a couple of MOOC’s on Data Science.</p><p>By this time, you must be pretty confident with the basic Data Analysis with Python. <br><em>Personal Suggestion: Do an intern at a startup as an Analyst or Engineer and get a basic understanding of how things work in the industry.</em></p><p>Okay, so back to the plan. Things are about to get real now. Where do you store all the Data? So as a Data Scientist you must know how to store, manipulate, aggregate, mine and handle Database. SQL, MongoDB, Cassandra, PostgreSQL and whatnot. Get a basic understanding of how storage works in a NoSQL environment.</p><p>Next comes Big Data — Hadoop &amp; MapReduce. Towards the end, you have to master Machine Learning and Deep Learning.</p><p>I’d suggest that throughout this plan, try to focus on one (or) two domains such as — eCommerce or Finance or Media or whatever that excites you. Do all the projects in that domain. Constantly learn more and more about that field. Domain is important because that will make you an expert and I don’t have to tell you that experts mint money far more than anyone else.</p><p>I know it feels overwhelming right now. But you have to do this over a year. So follow the plan, do one thing at a time and next year this time, you’ll be telling this story to your junior that how hard you worked to be a Data Scientist :)</p><p>Here the link to the original thread — <a href="http://bit.ly/2Bln4Ub">http://bit.ly/2Bln4Ub</a></p><p>PS: Next tutorial is coming up in a few days so stay around.</p><p><strong><em>Until then, keep hustling!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=181e838f698e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-weekly-dsw/data-science-plan-2018-181e838f698e">Data Science Plan 2018</a> was originally published in <a href="https://medium.com/data-science-weekly-dsw">Data Science Weekly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting MS Admission]]></title>
            <link>https://medium.com/data-science/predicting-ms-admission-afbad9c5c599?source=rss-a42d268507e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/afbad9c5c599</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Naman Doshi]]></dc:creator>
            <pubDate>Wed, 07 Feb 2018 07:35:10 GMT</pubDate>
            <atom:updated>2019-05-25T12:56:53.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f6SLb2T3N28aecjNIgACOQ.jpeg" /><figcaption>Credits: Pexels</figcaption></figure><p>It’s almost admission season and I’ve couple of friends who are in panic mode waiting for a call from the universities they’ve applied at.<br>This made me think — How can we predict whether a student will get an admit or not? What are the parameters for selection? Can it be mathematically expressed? <br>All of these questions started popping up, so I decided to work out this problem and help my friend calm his nerves. <br>(This is the first technical tutorial I’m ever writing so please go easy on me).</p><p>In order to solve this problem, I’m going to take you through what I call a<br> <strong><em>“Data Science Pipeline”. </em></strong>Given any problem statement, follow this pipeline so that your approach is structured.</p><ol><li><strong>Define the problem -</strong></li></ol><p>Write down the problem statement and understand what you’re trying to solve. In this case, our objective is to predict whether a student will get an admit or not. So it means, that this is a binary classification problem.</p><p><strong>2. Generate your own hypothesis -</strong></p><p>Next, list down all the things that you think can affect our objective viz. List down all the possible features with respect to our target feature.<br>In our case, we got ask this question — <em>What are the factors that can affect a student’s admission?<br></em>Take a piece of paper and write down your own hypothesis. I spent some time and came up with the following features<br>- GRE Score<br>- TOEFL Score<br>- Statement of Purpose (SOP)<br>- Letter of Recommendation (LOR)<br>- Academic Performance (GPA)<br>- Extra Curricular Activities (Sports, Math Olympiad etc..)<br>- Outstanding Achievements<br>- Projects and Research <br>Make sure that you write down at least 10–20 for any problem statement. This helps us get a deeper understand of what we are trying to solve and prompts us to think beyond the available dataset.</p><p><strong>3. Get the Dataset -</strong></p><p>Next step is to get the data. We’re using UCLA hypothetical data for graduate admissions. You can download the<a href="https://stats.idre.ucla.edu/stat/data/binary.csv"> dataset here</a>.<br>You can now map your hypothesis with the given dataset. See how many features you can add or modify in order to improve the quality of the dataset. Identify each feature as either <br>(a) Continous Variable or (b) Categorical Variable</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*gY6c3WDcUsIJDAKI_Rj-9w.gif" /><figcaption>(Source: Stack Exchange)</figcaption></figure><p><strong>4. Data Cleaning -</strong></p><p>Most of the time, the dataset will have lot of anomalies like missing values, outliers and so on. Always preprocess the data before moving on to the next step.<br>- <strong><em>Missing Value Treatment: </em></strong>You can use Mean-imputation (for continous variables) or Mode-imputation (for categorical variables)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*ZlLVuh-ikZ8so2fWL2m61w.png" /></figure><p>- <strong><em>Outlier treatment: </em></strong>An <em>outlier</em> is an observation that lies an abnormal distance from other values in a random sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*qF5JqaqfenZYB68pakIeUg.png" /></figure><p>There are four ways to treat it — Deleting observations, Imputing, Creating Bins and Treat Separately.<br>- <strong><em>Feature Engineering: </em></strong>Combining, adding, deleting, splitting, scaling features in order to increase the accuracy of the model</p><p><strong>5. Exploratory Data Analysis (EDA) -</strong></p><p>Now its time to get our hands dirty. Let us explore the data and understand the given features.<br>This dataset has a binary response (outcome, dependent) variable called admit. There are three predictor variables: gre, gpa and rank. We will treat the variables gre and gpa as continuous. The variable ranktakes on the values 1 through 4. Institutions with a rank of 1 have the highest prestige, while those with a rank of 4 have the lowest.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4dff009fa7b435a1387e974fb19841ab/href">https://medium.com/media/4dff009fa7b435a1387e974fb19841ab/href</a></iframe><p>EDA helps us get a clear idea about our features. It also helps us capture any trend or seasonality of data points.<br>In our case, we can see that Higher the GRE score and GPA, more the chances of getting admit.</p><p><strong>6. Predictive Modeling -<br></strong>This is a binary classification problem. The output has only two possibilities either <em>Yes (1) </em>or <em>No (0). </em>There are lot of classification algorithms out there so how do we know which one is the best?<br>We don’t. This is something that comes with experience and expertise. But not to worry, we do have a work around. We can fit multiple algorithms onto our data points and then evaluate the model. This way we can choose the best model which has the least error.<br>First, let us import all the necessary libraries</p><pre>from matplotlib import pyplot<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import classification_report<br>from sklearn.metrics import confusion_matrix<br>from sklearn.metrics import accuracy_score<br>from sklearn.model_selection import KFold<br>from sklearn.model_selection import cross_val_score<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.neighbors import KNeighborsClassifier<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.discriminant_analysis import LinearDiscriminantAnalysis<br>from sklearn.naive_bayes import GaussianNB<br>from sklearn.svm import SVC</pre><p>We’ll be using Cross Validation as an evaluation metric where in we will split our dataset into Train and Test.</p><pre><strong>#Step 1: Convert Dataframe into matrix</strong><br>dataArray = dataset.values</pre><pre><strong>#Step 2: Splitting Input features &amp; Output Variabl</strong>es<br>X = dataArray[:,1:4]<br>y = dataArray[:,0:1]</pre><pre><strong>#Step 3: Splitting training &amp; testing</strong><br>validation_size = 0.10<br>seed = 9<br>X_train, X_test, Y_train, Y_test = train_test_split(X,y,test_size=validation_size, random_state = seed)</pre><pre>print(X_train.shape)<br>print(X_test.shape)<br>print(Y_train.shape)<br>print(Y_test.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*ZeEzxXtGj3WNy6Yy13T_4Q.png" /></figure><p>Next we are going to apply the following Machine Learning algorithms on our Training data<br>- Logistic Regression (LR)<br>- Linear Discriminant Analysis (LDA)<br>- K Nearest Neighbours (KNN)<br>- Decision Tree (CART)<br>- Random Forest (RF)<br>- Gaussian Naive Bayes (NB)<br>- Support Vector Machines (SVM)<br>Initialize max number of features and number of trees.</p><pre>num_trees = 200<br>max_features = 3<br>models = []<br>models.append((&#39;LR&#39;, LogisticRegression()))<br>models.append((&#39;LDA&#39;, LinearDiscriminantAnalysis()))<br>models.append((&#39;KNN&#39;, KNeighborsClassifier()))<br>models.append((&#39;CART&#39;, DecisionTreeClassifier()))<br>models.append((&#39;RF&#39;, RandomForestClassifier(n_estimators=num_trees, max_features=max_features)))<br>models.append((&#39;NB&#39;, GaussianNB()))<br>models.append((&#39;SVM&#39;, SVC()))</pre><pre><strong>#Fit Models and Evaulate</strong></pre><pre>results = []<br>names = []<br>scoring = &#39;accuracy&#39;</pre><pre><strong>#Cross Validation</strong><br>for name, model in models:<br> kfold = KFold(n_splits = 10, random_state=7)<br> cv_results = cross_val_score(model, X_train, Y_train, cv=kfold, scoring = scoring)<br> results.append(cv_results)<br> names.append(name)<br> msg = &quot;%s: %f (%f)&quot; % (name,cv_results.mean(), cv_results.std())<br> print(msg)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*M88HOH_MBJushhcnSnJing.png" /></figure><p>What we have done here is run all 7 algorithms on our data point and tried to evaluate it. As you can see LR and LDA perform the best.<br>We can also implement a comparison plot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*-HfxfbORA6Y_UZgDAuy-FA.png" /></figure><p>So now we have fit the ML algorithm onto our dataset. Next, we have to create a predictive model using the best algorithm.<br>For the sake of simplicity, I’m going to pick LR.</p><pre><strong>#Step 1 - Create prediction model</strong><br>model = LogisticRegression()</pre><pre><strong>#Step 2 - Fit model</strong><br>model.fit(X_train, Y_train)</pre><pre><strong>#Step 3 - Predictions </strong><br>predictions = model.predict(X_test)</pre><pre><strong>#Step 4 - Check Accuracy</strong><br>print(&quot;Model --- LogisticRegression&quot;)<br>print(&quot;Accuracy: {} &quot;.format(accuracy_score(Y_test,predictions) * 100))<br>print(classification_report(Y_test, predictions))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*AxPw2UCa0OZSdVzN_7mhsQ.png" /></figure><p>I love to make plots. So lets make an another one ;)</p><pre><strong>#plotting confusion matrix on heatmap</strong><br>cm = confusion_matrix(Y_test, predictions)<br>sns.heatmap(cm, annot=True, xticklabels=[&#39;reject&#39;,&#39;admit&#39;], yticklabels=[&#39;reject&#39;,&#39;admit&#39;])<br>plt.figure(figsize=(3,3))<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*FRp99DF3nj1YgbtAT0JLbw.png" /></figure><p>Congratulations! You made our very own predictive model with an accuracy of 77.5<br>(Not bad at all for beginners)<br>You can try to increase this score by fitting different classification algorithms or by doing a little bit of feature engineering. <br>Best way is to use Ensembling methods, but that is for some other tutorial.</p><p>But wait…<br>How do we know that this works? <br>Yes, exactly right? We have come this far, how can we not let our model make some predictions!</p><pre><strong>#Making predictions on some new data<br>#Like a boss</strong><br>new_data = [(720,4,1), (300,2,3) , (400,3,4) ]</pre><pre><strong>#Convert to numpy array</strong><br>new_array = np.asarray(new_data)</pre><pre><strong>#Output Labels</strong><br>labels=[&quot;reject&quot;,&quot;admit&quot;]</pre><pre><strong>#Let&#39;s make some kickass predictions</strong><br>prediction=model.predict(new_array)</pre><pre>#G<strong>et number of test cases used</strong><br>no_of_test_cases, cols = new_array.shape</pre><pre>for i in range(no_of_test_cases):<br> print(&quot;Status of Student with GRE scores = {}, GPA grade = {}, Rank = {} will be ----- {}&quot;.format(new_data[i][0],new_data[i][1],new_data[i][2], labels[int(prediction[i])]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*Z9q2cCPb1v0Dwfw6FmYh0w.png" /></figure><p>AND THAT’S IT. Our very own predictive model that can decide your friend’s future (actually no, it cant)<br>But our model can definitely predict whether he/she will get an admit or not!</p><p><strong>Motives behind DSW </strong>—</p><p>I’ve been learning and doing data science from about a year now. During this time I have worked on lot of problem statements. But the thing is sometimes the dataset is pretty interesting but other times its not. You have to care about the problem you’re trying to solve, only then you’ll be able to do a good job.</p><p>So find a problem that you wanna solve and use data to solve it.</p><p>You can find the code for this tutorial<strong> </strong><a href="https://github.com/warmachine0609/Predicting_MS_admission"><strong>here on Github </strong></a>—</p><p>I’ll be writing down some kickass tutorials every week so if you’re someone who can learn only by doing, follow my <a href="https://medium.com/data-science-weekly-dsw/"><strong>publication here.</strong></a></p><p>You can also connect with on <a href="https://www.linkedin.com/in/naman-doshi/"><strong>LinkedIn</strong></a>.</p><p>I hope you have a productive week ahead.</p><p><strong><em>Until then, keep hustling!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afbad9c5c599" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/predicting-ms-admission-afbad9c5c599">Predicting MS Admission</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a environment for Data Science]]></title>
            <link>https://medium.com/data-science-weekly-dsw/setting-up-a-environment-for-data-science-44ca438ba36e?source=rss-a42d268507e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/44ca438ba36e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dsw]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Naman Doshi]]></dc:creator>
            <pubDate>Wed, 07 Feb 2018 05:37:40 GMT</pubDate>
            <atom:updated>2018-02-07T06:51:43.573Z</atom:updated>
            <content:encoded><![CDATA[<p>Now that you’ve decided that you want to get into this field, you might as well create an environment for your kickass future projects.<br>Personally, these are my system configurations / tools that I use for my projects and they work just fine.<br>- Hardware: HP notebook with i5(5th gen) processor, 8GB RAM<br>- OS: Windows 10/ Ubuntu 16.04 Dual Boot. <br>- Anaconda for Python programming<br>- RStudio for R Programming<br>- Sublime Text Editor<br>- Git (for Version Control)<br>- Tableau/ D3.js for Visualization<br>- Spotify (Need it!)</p><p>You can find a lot of materials online which will guide you through the installation process. I’ll just mention a few that I followed -</p><p>For Windows/ Mac you can follow<strong> </strong><a href="https://kfolds.com/setting-up-a-data-science-environment-5e6fd1cbd572"><strong>this guide</strong></a> by <a href="https://medium.com/u/2693d7c528a2">Danielle B</a><br>For Ubuntu, you can follow <a href="https://medium.com/@GalarnykMichael/install-python-on-ubuntu-anaconda-65623042cb5a"><strong>this guide</strong></a> by <a href="https://medium.com/u/c07aac64b6e1">Michael Galarnyk</a></p><p>Up next — We’ll be starting our first Data Science project that predicts whether a student will get an MS admit or not. So do check that out as well.</p><p><em>Until then, keep hustling!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44ca438ba36e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-weekly-dsw/setting-up-a-environment-for-data-science-44ca438ba36e">Setting up a environment for Data Science</a> was originally published in <a href="https://medium.com/data-science-weekly-dsw">Data Science Weekly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make Money while browsing Social Media!]]></title>
            <link>https://medium.com/@namandoshi/make-money-while-browsing-social-media-5683abf4adeb?source=rss-a42d268507e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5683abf4adeb</guid>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Naman Doshi]]></dc:creator>
            <pubDate>Fri, 08 Dec 2017 13:30:40 GMT</pubDate>
            <atom:updated>2017-12-08T13:30:40.698Z</atom:updated>
            <content:encoded><![CDATA[<p>What if I told you that you can make money while reading this post (or) scrolling through memes (or) just like/ commenting on a post?!</p><p>Sounds too good to be true? <br>Wondering how much money you would’ve made all these years?</p><p>Okay, I’ll tell you how you can do this.</p><p>But wait, in order to understand how this works, we have to understand few concepts. So let me tell you something about <strong><em>“Decentralized Applications”</em></strong> <br>I promise to not bug you. Just hear me out for a while.</p><p>The entire internet works on something called as <strong><em>“Client Server Architecture”</em></strong> which looks something like this —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*jLj4R-KeQbb8dCnb.png" /></figure><p>So basically, all of our data gets stored on a central server.<br>Consider this example — <br>You take out your phone and search for some mean Trump memes. What happens at this point is, your search request goes to Google’s server which then finds you good set of memes. Here only two entities are involved in the entire transaction.( I might have oversimplified things for the sake of understanding )</p><p>This is called as Web 2.0 but now we are moving towards a new age where there are no central servers. All the data is stored among the fellow peers on the network which means there are multiple entities involved. We are moving towards something rightly called as <strong>Web 3.0 </strong>(Tadaaa!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HoCsdMFxGc74gr8M.png" /></figure><p>Let me ask you this. Why do you think Google and Facebook and all the tech giants are giving you free service?</p><p>The cost of servers and keeping the platform up and running is humongous ! What are they getting in return?<br>(<strong>Hint</strong>: They are not doing social service!)</p><p><strong>Ans</strong>: YOUR DATA!</p><p>This is an era of technology and the biggest currency is DATA!</p><blockquote>A person who own’s your data owns you!</blockquote><p>So blamo! They sell your data to the advertisers and get paid!</p><p>But what if there was a way where we can cut the middle man out (Say Facebook in this scenario) then WE CAN GET PAID DIRECTLY BY THE ADVERTISERS/ CONTENT CREATORS! How friggin’ cool is that?</p><h3>Enter — Steemit.com</h3><p>Steemit is a decentralized social media platform where the content creators get paid for their posts/blogs. You get paid in Steemit’s currency called as the “Steem Power” which you can convert into USD or Bitcoin.<br>They use <strong><em>Blockchain</em></strong>, just like Bitcoins but here instead of saving a transaction, the network stores content.<br>The beauty of dApps (Decentralised apps) is that once you put something on the blockchain, nobody can take it down.<br>(I’m looking at you Kim Jong-un)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RpxIimXKbY4KfsEa.png" /></figure><p>You can see that how every post has a price based on the number on follows and likes.</p><h3>Let’s get you started —</h3><p>Step 1: Sign up <a href="https://steemit.com/">here!</a></p><p>Step 2: Since it is in a Beta stage, they’ll take 24 hours or less to go through your profile then provide you with a login</p><p>Step 3: You might wanna follow some people. Remember, even like or comment can fetch you money. For starters, you can <a href="https://steemit.com/@warmachine0609">follow me</a>. (I’m also a newbie, can use some friends over there)</p><p>Step 4: Write your first blog and make sure its entertaining. (it’s a social media after all, people love to see stupid things)</p><p>Step 5: Use appropriate Hash tags and you’re good to go</p><p>That is it. Next time, you might wanna spend less time on Facebook and more on Steemit ( &amp; fill your pocket )</p><p>*Happy Steeming*</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5683abf4adeb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>