<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aadit Kapoor on Medium]]></title>
        <description><![CDATA[Stories by Aadit Kapoor on Medium]]></description>
        <link>https://medium.com/@aaditkapoor?source=rss-39b97293486c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*S6fTwSUlzeqCMUFVpQq61g@2x.jpeg</url>
            <title>Stories by Aadit Kapoor on Medium</title>
            <link>https://medium.com/@aaditkapoor?source=rss-39b97293486c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:48:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aaditkapoor/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[To survive is to understand the nature of reality]]></title>
            <link>https://aaditkapoor.medium.com/to-survive-is-to-understand-the-nature-of-reality-a75a4696ae4f?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a75a4696ae4f</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[humanity]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[genomics]]></category>
            <category><![CDATA[world]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Fri, 28 Nov 2025 04:50:46 GMT</pubDate>
            <atom:updated>2025-11-28T04:56:23.008Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2023–24, I gave myself a simple challenge: treat my master’s program as an accelerated PhD. No classes-first, no “wait for the right advisor,” no incrementalism.<strong><em> Just 296 days out of 365 , about 80% of the year spent doing the only thing that mattered: understanding the most important field of our time, genomics.</em></strong></p><p>The routine looked more like an early-stage startup than a graduate degree.<br>Wake up.<br>Research.<br>Code.<br>Sleep.<br>Repeat.</p><p>That level of focus created a strange kind of clarity. When you compress years of exploration into months, you begin to see the contours of what truly matters. For me, the realization was simple but profound: like every civilization before us, humanity advances only when it gains a deeper understanding of reality. And for the first time in history, AI gives us a real shot at doing this at scale.</p><p>Not sci-fi.<br>Not philosophy.<br>A practical mechanism for understanding the systems that sustain human life.</p><p>AI is not just a tool — it’s an accelerant for human comprehension. It gives us leverage over complexity in a way nothing else ever has. Genomics is the clearest proof: billions of years of biological information, suddenly legible.</p><p>And that leads to the deeper idea.</p><p><strong>To survive is to understand reality.<br></strong>Civilizations don’t collapse because they run out of money, talent, or ambition.<br><strong>They collapse because they run out of time, because their rate of learning falls below the rate of external change.</strong></p><blockquote><em>Time, not capital, is the true limiting reagent of progress.</em></blockquote><p>Give any civilization enough time and clarity, and it will expand the frontier. Take that time away, and even the most advanced society will fail.</p><p>AI — especially AI applied to biology — is the first technology that meaningfully bends this curve. It compresses discovery cycles, shrinks research timelines, and expands what a single person can understand in a lifetime.</p><p>My year-long experiment taught me this:<br>When you remove everything except the pursuit of understanding, the world gets sharper, not smaller. And you begin to see how much more is possible when our constraints shift from human time to machine time.</p><p>The next era of progress will belong to those who internalize this shift early — those who treat understanding as the primary goal, and time as the scarcest resource.</p><h4><strong>Everything else is a rounding error.</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a75a4696ae4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I vibecoded a 3D Domino's Pizza Maker]]></title>
            <link>https://aaditkapoor.medium.com/how-i-vibecoded-a-3d-dominos-pizza-maker-0b3cb14fccc5?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b3cb14fccc5</guid>
            <category><![CDATA[llm-guardrails]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Mon, 11 Aug 2025 05:03:11 GMT</pubDate>
            <atom:updated>2025-08-11T05:15:34.636Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44v0gPB4LhTdot4mfVEikw.png" /></figure><h3>Introduction</h3><p>I’m a huge fan of Domino’s, but the current Pizza Maker interface just doesn’t cut it for me. With the rise of powerful AI coding tools, I took up the challenge to code a 3D Domino’s Pizza Maker from scratch, entirely powered by AI.</p><h3>What is Vibe Coding?</h3><p>Vibe Coding, popularized by Andrej Karpathy, is an AI-driven approach where code is generated by a large language model (LLM) based on prompts, streamlining and automating the coding process.</p><h3>Prompting Methodologies</h3><p>Since I am an Engineer, I was familiar with how I can design the system. Using a lightweight 3D engine such Three.js seemed sufficient. Prompting for vibe coding can be divided into the following methods</p><h4>Zero Shot</h4><p>Zero-shot prompting involves giving the AI a task without examples, relying on its pre-trained knowledge to generate an output</p><p><strong><em>“code a 3D dominos pizza maker simulation using three.js”</em></strong></p><p>This was zero-shot because the LLM inferred the requirements (e.g., toppings panel, 3D view, interactive elements)</p><h4>Few Shot</h4><p>Few-shot prompting provides a few examples in the prompt to guide the AI</p><p><strong>“drag around the pizza and more realistic visuals. Use OrbitControls if required”</strong></p><p>This was a few shot because the LLM used the provided examples such as building on it by adding OrbitControls for dragging and enhancing materials/shadows.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuclykHKqMgs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuclykHKqMgs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuclykHKqMgs%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e35a308ee9209ce578782274155d5e87/href">https://medium.com/media/e35a308ee9209ce578782274155d5e87/href</a></iframe><h3>Discussion</h3><p>Vibe coding, or automating the coding process, is here to stay. While having an engineering background currently helps in guiding AI more effectively, I believe that as AI capabilities advance, this advantage may diminish. What will remain highly valuable is the ability to craft precise, technical prompts. In particular, AI engineers who can design and write scripts that harness automation to streamline the coding process will be especially well-positioned to thrive</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b3cb14fccc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt Engineering: A new engineering craft]]></title>
            <link>https://medium.com/swlh/prompt-engineering-a-new-engineering-craft-863139f43e7c?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/863139f43e7c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[world]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Wed, 05 Apr 2023 22:12:43 GMT</pubDate>
            <atom:updated>2023-04-05T22:12:43.151Z</atom:updated>
            <content:encoded><![CDATA[<h4>Prompt Engineering: How would prompt engineering change the way people think?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_SQd6-Em1RPqCQ-A" /><figcaption>Photo by <a href="https://unsplash.com/it/@glenncarstenspeters?utm_source=medium&amp;utm_medium=referral">Glenn Carstens-Peters</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What is Prompt Engineering?</h3><p>Prompt Engineering is essentially the art to accurately extract information from an LLM (Large Language Models). I consider prompt engineering to be a broader job title (think of this as a data scientist). Prompt Engineering can be classified into the following categories:</p><ol><li><strong>Prompt Engineering (Non CS/ML background)</strong>: This category is targeted towards individuals who do not have a M.L background and are primarily people who would typically use an interface such as ChatGPT.</li><li><strong>Prompt Engineering (CS/ML background):</strong> This category is targetted towards individuals with a M.L background and are primarily Machine Learning Engineers/Data Scientists who can effectively combine prompt engineering methods and other M.L models to solve a data problem. Carefully designed prompts can easily be converted into a step by step process to solve a problem.</li></ol><blockquote>Prompt Engineering coupled with an additional skill would be what differentiates the ordinary prompt engineers and extraordinary prompt engineers.</blockquote><h3>Qualities for a good prompt engineer</h3><h4>A great prompt engineer comprises the following skills:</h4><ol><li>Preferred knowledge/experience/degree in CS/M.L/Data Science/A.I (Can hack things together etc)</li><li><strong>Domain expert in their field of choice</strong> (To be a strong prompt engineer, one needs to have an in-depth knowledge of the field)</li><li><strong>Critical Thinking and Scientific Temperament:</strong> Ability to question things scientifically and to ask good questions</li></ol><h3>Prompt Engineering Use cases</h3><p>Prompt Engineering can be used in any industry but what differentiates ordinary prompt engineers and extraordinary prompt engineers are machine learning engineering/full stack data science skills and the ability to hack things together and build a usable data product.</p><h3>Prompt Engineering as an augmented tool</h3><p>For practically any task, we can use prompt engineering to boost our productivity or ease up tasks. An example would be helping me write this blog.</p><h3>Prompt Engineering as a classifier</h3><p>Utilizing few-shot/one-shot prompting, prompt engineering can successfully decode complex classifications tasks. Combining these steps to solve a data business problem can be an important use case. Constructing business problems as a classification task is the most complex aspect of this methodology.</p><h3>Prompt Engineering in Computational Biomedicine</h3><p>Prompt Engineering in Medicine is no less than a revolutionary feat but at this point in time, the systems are not as robust. Prompt Engineering in Biomedicine presents a new paradigm where medical doctors would just “prompt” the medical LLM to specify steps, treatments, send notifications, etc. It would be like an intelligent medical operating system but unlike a regular OS, this would mostly be chat driven. Hallucination is the major concern for these LLMs and employing domain experts would allow us to reduce hallucination instances, particularly in medicine.</p><h3>Prompt Engineering for design</h3><p>A basic level of Prompt Engineering would be the goto standard for designing graphics. A good graphics designer would be someone would is an expert in describing the design not using the tools. This highlights the biggest change Prompt Engineering would create, <strong>critical thinking.</strong></p><h3>Conclusion</h3><p>Prompt Engineering defines a new paradigm and as A.I systems become more advanced, we will see an increased use of the word “prompt engineer”. Increaseing depth instead of breadth would be extremely valuable and would practically change how things take place.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=863139f43e7c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/prompt-engineering-a-new-engineering-craft-863139f43e7c">Prompt Engineering: A new engineering craft</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use Data Science to predict if an H1B petition would be certified, withdrawn, or denied?]]></title>
            <link>https://aaditkapoor.medium.com/how-to-use-data-science-to-predict-if-an-h1b-petition-would-be-certified-withdrawn-or-denied-bd1a2ea47e9f?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd1a2ea47e9f</guid>
            <category><![CDATA[work]]></category>
            <category><![CDATA[visa]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Wed, 24 Aug 2022 07:04:51 GMT</pubDate>
            <atom:updated>2022-08-24T07:04:51.407Z</atom:updated>
            <content:encoded><![CDATA[<h4>This case study explores the application of Data Science to predict if an H1B petition is certified, withdrawn, or denied.</h4><blockquote>This case study was brought to you by Case Studies @ Data Kapoor™. Explore one click implementation catered to your use case only at Case Studies @ Data Kapoor.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V2pWridRZKsTenUx" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>H1B is the most popular visa in the United States for long-term work options. H1B nonimmigrant visas allows employers to petition for highly educated foreign professionals to work in “specialty occupations” that require at least a bachelor’s degree or the equivalent.</p><p>To obtain an H1B Visa, an employer has to follow two steps (More information <a href="https://www.americanimmigrationcouncil.org/research/h1b-visa-program-fact-sheet">here</a>):</p><ul><li>Employers first must attest, on a labor condition application (LCA) certified by the Department of Labor (DOL), that employment of the H-1B worker will not adversely affect the wages and working conditions of similarly employed U.S. workers</li><li>Employers must also provide existing workers with notice of their intention to hire an H-1B worker.</li></ul><h3>Problem Statement</h3><p>Each fiscal year, USCIS releases LCAs of H1B petitions with information about the certification or denial of an application. The probability of an H1B petition being picked is estimated at <strong>less than 38% (</strong><a href="https://www.lawfirm4immigrants.com/what-h-1b-lottery-results-says-about-odds/#:~:text=During%20the%20H%2D1B%20Cap%202020%20filing%20period%20USCIS%20received,to%20the%20advanced%20degree%20exemption).&amp;text=In%202018%20(fy%202019)%20170%2C098,being%20selected%20for%20H1B%20visa."><strong>Reference</strong></a><strong>).</strong></p><blockquote>The current annual statutory cap is 65,000 visas, with 20,000 additional visas for foreign professionals who graduate with a master’s degree or doctorate from a U.S. institution of higher learning — <a href="https://www.americanimmigrationcouncil.org/research/h1b-visa-program-fact-sheet">Reference</a></blockquote><p>While there is no control over the lottery, we seek to tackle the prediction of H1B petition as certified, denied, or withdrawn as a classification problem.</p><p>We aim to find relations or combinations of features (columns or H1B petition data) that lead to a case into any of three categories.</p><h3>Tools Required</h3><ul><li>Python 3</li><li>PySpark</li><li>Pandas</li><li>Seaborn</li><li>Scikit-Learn</li><li>NumPy</li></ul><h3>Data Sources</h3><p>We use H1B visa petitions from 2011–2016 (<a href="https://www.kaggle.com/datasets/nsharan/h-1b-visa">Data Source</a>). Following is the screenshot of the dataset with the following features:</p><ul><li><strong>CASE_STATUS</strong>: The case status</li><li><strong>EMPLOYER_NAME</strong>: Employer Name</li><li><strong>SOC_NAME</strong>: Job Title Category</li><li><strong>JOB_TITLE</strong>: Job Title</li><li><strong>FULL_TIME_POSITION</strong>: Is this a full-time position</li><li><strong>PREVAILING_WAG</strong>E: The prevailing wage</li><li><strong>YEAR 0 WORKSITE</strong>: Work location</li><li><strong>lon</strong>: Longitute</li><li><strong>lat</strong>: Latitude</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0TU0S9KYzJ_1YOkOY8nN5Q.png" /><figcaption>Dataset Image</figcaption></figure><h3>Exploratory Data Analysis</h3><p>The dataset contains 2,895,144 rows where the years range from 2011 to 2016. While the dataset does not contain much information that also contributes to the factors of H1B selection (Such as degree level) and we also do not the how the lottery proceeds, we can still get a lot of information by first preprocessing the data.</p><h4>Data Preprocessing</h4><p>The dataset contains a lot of categorical columns. To do any analysis with respect to year, we will first perform label encoding.</p><p>For the columns:</p><ul><li><strong>CASE_STATUS</strong></li><li><strong>EMPLOYER_NAME</strong></li><li><strong>SOC_NAME</strong></li><li><strong>JOB_TITLE</strong></li><li><strong>FULL_TIME_POSITION</strong></li><li><strong>WORKSITE</strong></li></ul><p>Additionally, we will store the label encoder object and the corresponding mapping in a dictionary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*Pv0glLiF-YOz4imc1aH-wA.png" /><figcaption>Demonstration of the label encoding for one of the columns</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H5M1Y6edPwCQv6gciK5HqA.png" /><figcaption>Final Dataset</figcaption></figure><p>First, let’s check for any class imbalances. As you can see labels (4, 3, 5) <strong>ENDING QUALITY AND COMPLIANCE REVIEW — UNASSIGNED, INVALIDATED, REJECTED </strong>are extremely few. To handle these cases is out of scope for this case study.</p><blockquote><strong>Guess what? </strong><a href="https://casestudies.datakapoor.com/"><strong>Case Studies @ Data Kapoor™</strong></a><strong> frequently manages such client cases/use cases where there is major class imbalance issue. If you have a similar use case and are struggling, we are always there for you!</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*m2xklVq2gNPw86qobU54Qg.png" /><figcaption>Class Count</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*kVtOAVvmb3-FvjexSpEg7Q.png" /><figcaption>The prevailing wage has gone down as the years proceed. Perhaps, we can categorize this phenomenon as the abuse of H1B by certain companies.</figcaption></figure><p>If we see the line plots of how CASE_STATUS has changed over the years, we see a clear trend in the popularity of the H1B visa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*kNUg8AyyyJL60xOzq3s0rg.png" /><figcaption>Over the years, we also see the CASE_STATUS (Certified) has gone up</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*-B0EC3TIPbW4EZdstaBHNg.png" /><figcaption>Over the years, we also see the CASE_STATUS (Denied) has gone down. For CASE STATUS (Rejected), we only have one case.</figcaption></figure><blockquote>There is a difference between “rejection” and “denial” in the immigration world. A rejection simply means that there was an error with your filing or fee payment that can be corrected. A denial occurs when either you or your employer are not considered qualified for an H-1B — <a href="https://www.immi-usa.com/h1b-visa/denial/">Reference</a></blockquote><p>Additionally, the most popular job title with the highest number of petitions is “programmer analyst”.</p><blockquote>By performing label encoding, we were able to analyze categorically to highlight key trends and most importantly understand the data</blockquote><h3>Prediction Models</h3><p>To start off, we will be evaluating two models:</p><ul><li>Random Forest</li><li>XGBoost</li></ul><h4>Feature Selection</h4><p>We currently have 9 features (columns). To reduce the complexity and to identify relevant columns, we use Mutual Information. We use sklearn’s SelectKBest with the <em>score_func as mutual information (or </em>mutual_info_classif in sklearn). We select the best 3 features making our shape as follows:</p><pre>(2895144, 9) -&gt; (2895144, 3)</pre><h4>Train, Test, Validation Splits</h4><p>We split the dataset into 60% (train), 20% (test), 20% (validation).</p><p>Following are the metrics:</p><h4>Random Forest</h4><p>Training Accuracy:<strong> 90%</strong></p><p>Testing Accuracy: <strong>86%</strong></p><p>Validation Accuracy: <strong>86%</strong></p><h4>Note: We do have slight overfitting</h4><h4>XGBoost</h4><p>Training Accuracy:<strong> 87%</strong></p><p>Testing Accuracy: <strong>87%</strong></p><p>Validation Accuracy: <strong>87%</strong></p><h4>Note: There is no overfitting</h4><p>Based on this brief analysis, we can infer that XGBoost is the best prediction model.</p><h3>Case Study Observations and inferences</h3><p>We have successfully developed a good model to predict H1B case status. With 87% testing accuracy we have achieved great results by using mutual information feature selection.</p><h4>Why is this prediction problem relevant?</h4><p>For this year, we had 483,927 H1B petitions filed. The relevance of this prediction is extremely apt given the current scenario. Anyone on a work visa faces extreme anxiety and frustration when they are eagerly waiting for the next steps.</p><blockquote><strong>This prediction problem solves the issue of anticipation.</strong></blockquote><p>Using this prediction model, we can predict the case status given the petition columns and potentially determine if there would be a second lottery in FY 2023. Additionally, we can also find out anomalous behaviors and extract insights into how the H1B program is being utilized.</p><h4><strong>Code</strong></h4><p><strong>Access Notebook: </strong><a href="https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1357612241790951/1729549686607869/585007093667772/latest.html"><strong>Notebook</strong></a></p><blockquote>This case study was brought to you by Case Studies @ Data Kapoor™. Explore one click implementation catered to your use case only at Case Studies @ Data Kapoor.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd1a2ea47e9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What did I learn while working as a data scientist?]]></title>
            <link>https://medium.com/swlh/what-did-i-learn-working-as-a-data-scientist-90e03f8ab9e6?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/90e03f8ab9e6</guid>
            <category><![CDATA[wall-st]]></category>
            <category><![CDATA[business-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Tue, 09 Aug 2022 03:42:29 GMT</pubDate>
            <atom:updated>2022-08-09T03:47:01.076Z</atom:updated>
            <content:encoded><![CDATA[<h4>My experience as a data scientist at a large startup. Why is real-world A.I/M.L so different from academia? How do we appropriately integrate business and science/engineering?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g7z83pdJSI04ef4j" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Learning Phase One</h3><p>I worked as a data scientist at <a href="http://swiggy.com/">Swiggy</a> for 1 year, a large Indian food delivery startup (around 20 million active users per month). Unlike other teams (such as recommendations, etc) where if the prediction is a bit off then the teams can quickly look into and push changes, I worked in the logistics team where every second was essential. I enjoyed every second of the intense and highly rewarding work. I would also like to thank my mentors for believing in me and guiding me at every step possible.</p><p>With all the innovations happening in data science/A.I field almost every day, one can lose sight of why is it so difficult to extract practical value from a wonderfully written research paper. This is not just valid for the data science field but almost any field.</p><blockquote>A.I/M.L is extremely fuzzy with its two driving pillars, engineering, and science. A.I is almost identical to biology, while it seems extremely simple, the processes occur in a highly complex and efficient manner.</blockquote><p>At Swiggy, I realized the importance of business value in data science where every second was essential. I realized what it means to be simple but complex. Understanding the business problem was 10x more important than deciding on the algorithm. While we could have applied any advanced algorithm, we needed explainability and robustness. Thinking over the business problem may sound boring but to truly harness the power of data science and Artificial Intelligence understanding the data is absolutely critical.</p><blockquote><strong>If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and five minutes thinking about solutions</strong>.</blockquote><blockquote>- Einstein</blockquote><h3>Learning Phase Two</h3><p>In my next Data Science and Artificial intelligence endeavor, I worked for a medical dental company for about 1.5 years where I truly became one with the data. With a huge influx of data collected by the machines, my job was to implement algorithms to better understand/predict machine failures. To understand the data is to accurately identify the use cases by asking the right questions. Asking the right questions is 100 times more difficult than finding the correct answer.</p><blockquote>A lot of times we are made to think that business and engineering/science are two different things but in reality and particularly in this field, they are the same thing. It is as if it is the same coin with one side being business and the other side being science.</blockquote><h3>Learning Phase Three</h3><p>I also worked for a number of research labs during my undergrad <strong>(I just graduated this May!</strong>). Unlike the industry, there was a clear distinction where the main aim of the project was to find the right algorithm for the already cleaned and defined data. Particularly this involved beating a specified benchmark. This is extremely different from the industry as a lot of times we encounter data drift, i.e the model that was getting a substantial accuracy does not work for a production environment.</p><h3>Conclusion</h3><p>Data Science or A.I is not about having a Ph.D. or being the smartest person. It’s about asking the right questions and staying humble. Particularly A.I is seen as an engineering science where implementing ideas is just as important as inventing them. No matter what anyone tells you, DS is not glamorous, it is a lot of applied grunt work with a focused business goal. Implementing these techniques in your business requires a lot more than building a complex M.L model.</p><h3>Need Help?</h3><p>Need help turbocharging your business with cutting-edge A.I/data science research in a practical way? Data Kapoor is there for you! We welcome you to become part of our 7+ happy customers!</p><p>Register Interest: <a href="https://www.datakapoor.com/">https://www.datakapoor.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90e03f8ab9e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/what-did-i-learn-working-as-a-data-scientist-90e03f8ab9e6">What did I learn while working as a data scientist?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What exactly is Data Science?]]></title>
            <link>https://medium.com/swlh/what-exactly-is-data-science-1dbdd259d0e?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dbdd259d0e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[problem-solving]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Thu, 27 Jan 2022 19:43:16 GMT</pubDate>
            <atom:updated>2022-01-27T19:43:16.066Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3G9O-HihFAC8TlOpjYCTEQ.jpeg" /><figcaption><a href="https://www.dreamstime.com/stock-illustration-problem-solving-diagram-question-marks-brain-lightbulb-to-illustrate-problems-solutions-image43014958">https://www.dreamstime.com/stock-illustration-problem-solving-diagram-question-marks-brain-lightbulb-to-illustrate-problems-solutions-image43014958</a></figcaption></figure><h3>Introduction</h3><p>Professionally being in the field of Data Science for a year (Well, humans are natural data scientists) I have seen and examined a lot of different perceptions about the field. If we think of Data Science then we see that there is a certain hype around the positions available in the market. A lot of us do not yet understand the true meaning of Data Science. I have seen that a lot of companies also are misguided and a lot of my peers also give in to that hype.</p><p>Well, there is indeed hype around Data Science. According to <a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">Harvard Business Review,</a> Data Scientist is the sexiest job of the 21st century. Data Science is a very interdisciplinary field with a lot of applications in logistics, recommendation systems, entertainment, etc. Essentially, Data Science can be seen in every industry one way or another.</p><p>In this article, I would like to give my views on what I think Data Science is and most importantly why it has always been an important problem tool. I would show my readers how instead of a field, we can think of Data Science as an art and a way of thinking. With this, I would like to define a new term known as data science thinking which is a combination of first principle and second order thinking.</p><h3>Assumptions</h3><p>To start let us base this article on an assumption that when I talk about Data Science, I encapsulate all the available roles in the market such as:</p><ul><li>Data Analysts</li><li>Data Scientists (Applied and Analytics)</li><li>Data Curator/Data Annotator</li><li>A.I Research Scientist</li><li>ML Engineers/A.I Engineers/Analytics Engineer</li></ul><p>Another assumption that we are considering is that we’d be looking at Data Science from a problem solving perspective as opposed to an educational perspective.</p><h3>What is Data Science?</h3><p>Data Science is an art but more specifically all data scientists or a person doing data science is a problem solver. On a wider spectrum, problems can be defined as of two types:</p><ol><li>Real World Problem: A real world problem is a problem that is solved to directly impact an end-user. These problems are highly efficient and they have to be scalable. These problems generally do not require new ideas or new formulas but they may require a new way of solving things or in data science, this is known as applied data science. (Examples include, Using data science techniques to increase grocery sales or using data science to increase the speed of an algorithm). These problems require technical as well as business skills.</li><li>Research/Scientific Problem: A scientific problem is a problem that is solved to prove something. Typically, these problems do not have a direct business impact and they do not always solve a business problem. A lot of times, these problems once solved have to be reiterated to be able to put the technology in use. These problems are not always scalable and efficient. (Examples include, a novel algorithm, A new type of Neural Network, a formula). These problems require extremely good technical skills.</li></ol><p>A data scientist is a problem solver that could be working to solve any one of the problems or a combination of the two. A data scientist’s job is to increase impact in an efficient and scalable manner.</p><h3>Highly valuable skills for a problem solver</h3><p>Apart from technical skills, an extremely good data scientist needs the following attributes:</p><ul><li>Curiosity: A data scientist should be extremely curious and should always seek on-demand knowledge</li><li>Passion: A extreme passion for the field</li><li>Critical Thinking: An ability to critically analyze and asks questions. If the question is correct then the answer is always correct</li><li>Business or Domain Knowledge: Business knowledge in the field they are solving</li><li>Organizational skills: Good organization skills. Good command on structuring notes, code, data, etc.</li><li>Humility: Always being humble</li><li>A focus on failing fast and reiteration and not being obsessed with perfection: Not to focus on perfection but to try to fail fast (without any fear) and trying new and bold things</li><li>Second Order Thinking: Ability to seek “then what” for a given situation</li></ul><blockquote><em>While a good command of technical skills is extremely valuable, those skills can be taught but non-technical (as listed above) is difficult to teach.</em></blockquote><h3>How to Adopt the “Data Science Thinking”?</h3><p>Data Science Thinking is a combination of second order and first principle thinking. In simple, what we do is basically break down a problem to its first principle and then for each subset of the problem, we apply second order thinking. This step ensures that we put extra emphasis on the questions.</p><blockquote><em>“If I had an hour to solve a problem and my life depended on the solution, I would spend the first 55 minutes determining the proper question to ask… for once I know the proper question, I could solve the problem in less than five minutes.” — Einstein</em></blockquote><p>This thinking also helps us to become better problem solvers. While I always used first and second-order thinking, I evaluated the use cases for the combination of the two.</p><h3>What is Data Science? — According to me and what I think would be extremely beneficial for the universe</h3><p>Data Science to me is all about solving real world problems in a scaleable manner. It is about adopting “data science thinking”. To me, Data Science is all about the business impact and solving new, difficult problems in a creative manner.</p><p>To me, failing fast and reiterating is extremely important because that is how we humans innovate. Imagination is something that makes us human and we should use that to solve real-world problems quickly.</p><h3>Solving a Data Science problem using “Data Science Thinking”</h3><p>Let us solve a data science (real world problem) using “Data Science Thinking”.</p><p>Problem Statement: What defines a popular song?</p><p>Available Data/Research: Given the problem statement, our job is to find out a way to quantitatively define what makes a song popular. Assuming, we have the data available, we first break down the problem into several subproblems:</p><p>Sub Problems:</p><ol><li>What is a Song?</li><li>How to define a song?</li><li>How to define a popular song? (Number of views, singer, lyrics, etc)</li><li>Does the song consist of lyrics or is it just a tune?</li><li>What is the platform of origin?</li><li>Why do people play a song again and again? (Motivation, lyrics, or emotions)</li><li>What is the emotional setting?</li></ol><p>Data Science thinking involves asking a lot of questions to yourself from a first principle and as second order thinking methodology.</p><p>This is a similar process where we identify the most important parts of a story or movie and then make an effort to understand it.</p><p>From a data science perspective, we have to optimize on the most generic solution and this involves second-order thinking of asking the question that seeks “then what” of a given subproblem.</p><p>This is an example of how we can use Data Science thinking to solve a particular problem. Note that there may be many variations but the idea is to ask a lot of questions and narrow down to the most encapsulated one while performing second-order thinking.</p><h3>Conclusion</h3><p>We examined what exactly is Data Science and more specifically, we divided problems into two categories. While Data Science can be broad, the maximum benefit that could help push the human race forward is building solutions for real-world problems (Typically using or utilizing research papers/solutions, etc) in a creative manner. This requires an attitude of failing fast and reiteration with a clear focus on the business impact/metric.</p><p>Linkedin: <a href="https://www.linkedin.com/in/aaditkapoor1201/">https://www.linkedin.com/in/aaditkapoor1201/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dbdd259d0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/what-exactly-is-data-science-1dbdd259d0e">What exactly is Data Science?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CovidSimNet: Introducing a Method to Determine “Similar” COVID States]]></title>
            <link>https://medium.com/data-science/covidsimnet-introducing-a-method-to-determine-similar-covid-cities-6d6701aad03b?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d6701aad03b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Thu, 15 Jul 2021 08:46:26 GMT</pubDate>
            <atom:updated>2021-07-20T05:13:13.991Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using data to determine a COVID state similarity matrix to estimate policy effects on similar states and “high risk” states</h4><p>Data has always played an integral part in analyzing and proving hypotheses. With the advent of highly optimized and easy-to-use frameworks, data is collected almost every second. The world’s most valuable resource is no longer oil, but data (<a href="https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data">The Economist</a>). It is estimated that by 2025, about 463 exabytes (one exabyte holds 50,000 years of DVD quality video) of data would be generated each day.</p><p><a href="https://www.rcseng.ac.uk/library-and-publications/library/blog/mapping-disease-john-snow-and-cholera/">John Snow</a>, an English physician and one of the founders of modern epidemiology, in the 1800s discovered the source of Cholera was contaminated public water pump. He discovered this by plotting cases of Cholera on a map and investigated that the majority of cases were of people close to the water pump.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q08imrzsIiX8JpA7" /><figcaption>Photo by <a href="https://unsplash.com/@mbaumi?utm_source=medium&amp;utm_medium=referral">Mika Baumeister</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>In this article, we present a plausible method that can be used to identify similar COVID-hit states by creating a state-by-state similarity matrix. This matrix can further be used for estimating policy effects for many states at once and also for determining high-risk states.</p><h3>The Data</h3><p>We will be using the COVID data for each US state comprising of confirmed cases, deaths, and the population for each state. We retrieve the dataset from <a href="https://github.com/CSSEGISandData/COVID-19"><strong>COVID-19 Data Repository by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLJRhUH1q5G2Q4aCejGHqw.png" /><figcaption>For each state and date, we have the confirmed cases (similarly for deaths) (Image by Author)</figcaption></figure><p>To extract the relevant information from these Pandas DataFrame, we first aggregate the confirmed numbers and death numbers for each date and then merge it with the population for each state. Upon performing all the preprocessing and aggregations, we retrieve the final DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*sRmVuPHhR9y2C9Rew0qGOA.png" /><figcaption>Our final DataFrame (Image by Author)</figcaption></figure><p>Now we derive the normalized quantities for the given columns by scaling the values between 0 and 1. We use the following formula and the accompanying code to compute the required values</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mwUo9OFTeAlt8GJ0.png" /><figcaption>Image by Author</figcaption></figure><pre>final[‘death_normalized’] = (final[‘total_deaths’] — final[‘total_deaths’].min()) / (final[‘total_deaths’].max() — final[‘total_deaths’].min())</pre><pre>final[‘confirmed_normalized’] = (final[‘total_confirmed’] — final[‘total_confirmed’].min()) / (final[‘total_confirmed’].max() — final[‘total_confirmed’].min())</pre><pre>final[‘population_normalized’] = (final[‘Population’] — final[‘Population’].min()) / (final[‘Population’].max() — final[‘Population’].min())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O3Ll_Xfk8EQYEGYEcJ3nZQ.png" /><figcaption>The normalized DataFrame (Image by Author)</figcaption></figure><blockquote><strong>Note</strong>: To use the death term and the confirmed term, we add the two quantities so as to get a term that quantifies information for both of the columns. This is a subjective decision and can be catered to how the objective is set.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*XtU9r2PyTSxkEdwB4nIlIA.png" /><figcaption>We further get the minimum between 1 and the total to limit the maximum value to 1 (Image by Author)</figcaption></figure><pre>def get_min(row): return min(1, row[&#39;total&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cwbXI0-i67bYoZ554ZsSWw.png" /><figcaption>Computed DataFrame (Image by Author)</figcaption></figure><h3>Formulation of the Metric</h3><p>We define the metric as the minimum between 1 and the addition of the normalized columns per population. For each state, we divide the addition of the normalized columns by the population. For a state, “risk factor” denotes how “risky” the state is. We use the following code and formula to compute the metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*yHtvxBh__KY_pyXFSiNhCg.png" /><figcaption>Risk Factor Formula (Image by Author)</figcaption></figure><pre>final[‘risk_factor’]=final[‘combined’]/final[‘population_normalized]<br>final = final.replace(np.inf, 0)<br>final[‘risk_factor_softmax’] = softmax(final[‘risk_factor’])</pre><blockquote><strong>Note</strong>: that this is a subjective metric and can be tweaked according to the objective. Alternate methods include get the population per total or using the maximum population.</blockquote><h4>Introduction to Softmax</h4><p>Upon computing the “risk factor”, we can see that the values are not scaled between 0 or 1. We want this as probabilities such that we can build a co-occurrence matrix for each state that represents similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/1*bvmmb0_wFG8nZGUncdOopw.png" /><figcaption>Risk Factor values are not between 0 and 1 (Image by Author)</figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Softmax_function">The softmax function</a> normalizes the given vector into a probability distribution such that each component will be in between 0 and 1 and the components will add up to 1 so that they can be interpreted as probabilities, the larger input components will correspond to larger probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*wi5g5f6uEMZRm-zgwTWmuQ.png" /><figcaption>Softmax Function (Image by Author)</figcaption></figure><p>Now to convert our values as probabilities and ultimately create a co-occurrence matrix of states, we apply softmax to the vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*792lyaGmJPPs5k2VCrgncg.png" /><figcaption>Softmax Applied (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*ctMXGHwgkvtxH5nFHN76ng.png" /><figcaption>Final DataFrame (Image by Author)</figcaption></figure><h3>Defining the “High Risk Area”</h3><p>A high risk area is defined as the state that exceeds the tolerance factor. If a state is equal or greater than the “tolerance factor”, we classify that state as a “high risk area”. For our current task, we set the “tolerance factor” as <strong>0.02 but this can vary.</strong></p><blockquote>Note: The Tolerance is based on the values and can change for different data (For example, if instead of the US, we are analyzing India’s COVID data).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*Zn_Cq5ZRBZ4RzY68H58ysQ.png" /><figcaption>A detailed algorithmic expression for determining high risk areas (Image by Author)</figcaption></figure><pre>tolerance_factor = 0.02</pre><pre>def is_risk_area(row):</pre><pre>   if row[&#39;risk_factor_softmax&#39;] &gt;= tolerance_factor:</pre><pre>     return True</pre><pre>   else:</pre><pre>     return False</pre><pre>final[&#39;is_risk&#39;] = final.apply(is_risk_area, axis=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*qr2oR2opP_y_UzTrWkBpUg.png" /><figcaption>A plot representing the counts of “high risk” areas (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*ZOls9-lCNfMGDpDY20B_ig.png" /><figcaption>DataFrame consisting of the transformed columns (Image by Author)</figcaption></figure><h3>Building the co-occurrence matrix a.k.a similarity matrix</h3><p>To calculate how similar are two states, in terms of “risk factor”, we define a term called “difference factor”. A difference factor represents the minimum difference required for the states to be considered similar. For our current task, the difference factor was set as 0.01.</p><blockquote>Note: The Difference factor is based on the values and can change for different data (For example, if instead of the US, we are analyzing India’s COVID data). The metrics defined are subjective and can be tweaked according to the different preferences.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*PZGDxlB0KQpAB1Q3RV0Eqg.png" /><figcaption>A detailed algorithmic expression for determining similarity (Image by Author)</figcaption></figure><blockquote><strong>Tip</strong>: To extract a more thorough similarity score, we can return the complement of the probability such that we have values between 0 and 1 representing the strength between two states. We can also include other factors as well.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*H6uatcQvIp5-e8Pi6cE1Mw.png" /><figcaption>Computed DataFrame (Image by Author)</figcaption></figure><h4>Deriving the co-occurrence matrix</h4><p>To derive the co-occurrence matrix from the above DataFrame, we create a coo_matrix using the scipy.sparse module. This function will create a matrix of dimension (number of states, number of states) i.e (58, 58) in our case. For each state, we would have the score corresponding to another state.</p><pre>from scipy import sparse<br>user_items = sparse.coo_matrix((cmat.score.astype(float),(cmat.state1.astype(&#39;category&#39;).cat.codes,<br>cmat.state2.astype(&#39;category&#39;).cat.codes)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*ik6uvgBGsmzO6X1D7MlfVQ.png" /><figcaption>Sample of the final DataFrame (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZYcvvlE8gPx3-lqxnGxew.png" /><figcaption>A heatmap representing how similar each state is to another state based on the confirmed and death numbers (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*PpsVym3EaCOOMbQrewfsAg.png" /><figcaption>An example of how we see if two states are similar (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*WKdFPWiA8oyrTv1Upjrh9g.png" /><figcaption>Demo output for two similar states (based on the risk factor) (Image by Author)</figcaption></figure><h3>Conclusion and Further Applications</h3><p>We demonstrate a plausible method to determine “high risk” states and build a map of similar states. The metrics defined are subjective and can be tweaked on the basis of the objective. The applications of the method are limitless as this can be applied to:</p><ul><li><strong><em>Estimating policy effect for states. (For example, determining the effect of closing restaurants on weekends for a state and approximating effect on another using the co-occurrence similarity matrix)</em></strong></li><li><strong><em>Detecting High Risk states or cities</em></strong></li></ul><p><em>and a lot more….</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d6701aad03b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/covidsimnet-introducing-a-method-to-determine-similar-covid-cities-6d6701aad03b">CovidSimNet: Introducing a Method to Determine “Similar” COVID States</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding Common Ground: US Presidents State Analysis]]></title>
            <link>https://pub.towardsai.net/finding-common-ground-us-presidents-state-analysis-cdd20cc22086?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdd20cc22086</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Sat, 28 Nov 2020 19:18:17 GMT</pubDate>
            <atom:updated>2020-11-29T23:10:15.373Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/data-science">Data Science</a>, <a href="https://towardsai.net/p/category/data-visualization">Data Visualization</a></h4><h4>Finding common ground: US Presidents State Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*JHqY149F8Io85nbH" /><figcaption><a href="https://unsplash.com/photos/zfKlCKK-Ql0">https://unsplash.com/photos/zfKlCKK-Ql0</a></figcaption></figure><h3>Introduction</h3><p>We aim to find any correlation between a US President and its state concerning other Presidents. We use data from different sources to form a dataset that comprises different US presidents and their home states with additional facts about the state. We pull data from multiple sources and combine it to form a comprehensive dataset. We also find the most common state and ultimately plot the results on a live interactive map.</p><h3>Requirements</h3><ul><li>Python 3.8</li><li>Pandas</li><li>NumPy</li><li>Folium</li><li>Seaborn</li></ul><h3>Data Preparation and Preprocessing</h3><p>We start by preparing the data to comb the datasets from different sources and perform some basic preprocessing on it.</p><ul><li>We change the types</li><li>Removed quotes and spaces</li><li>Changed columns and combined two CSV files</li></ul><pre><strong>def</strong> load_data(path:str<strong>=</strong>&quot;/&quot;):<br>    &quot;&quot;&quot;<br>        Loads the specified data using the path<br><br>        Args<br>        -------<br>        path: str (where president_timelines.csv and president_states.csv reside)<br><br>        Returns<br>        --------<br>        The combined dataframe<br>    &quot;&quot;&quot;<br>    df_time <strong>=</strong> pd<strong>.</strong>read_csv(&quot;president_timelines.csv&quot;)<br>    df_states <strong>=</strong> pd<strong>.</strong>read_csv(&quot;president_states.csv&quot;)<br>    <em># Setting columns as specified by the dataframe</em><br>    df_time<strong>.</strong>columns <strong>=</strong> [&quot;Index&quot;, &quot;Name&quot;, &quot;Birth&quot;, &quot;Death&quot;, &quot;TermBegin&quot;, &quot;TermEnd&quot;]<br>    df_states<strong>.</strong>columns <strong>=</strong> [&quot;Name&quot;, &quot;Birth State&quot;]<br>    president_names <strong>=</strong> df_time<strong>.</strong>Name<br>    print(f&quot;Total number of presidents: {len(president_names)}&quot;)<br>    president_states <strong>=</strong> df_states[&quot;Birth State&quot;]<br>    print(f&quot;Total states: {len(president_states)}&quot;)<br>    df <strong>=</strong> pd<strong>.</strong>DataFrame({&quot;name&quot;:president_names, &quot;states&quot;:president_states})<br>    <em># changing type</em><br>    df<strong>.</strong>name <strong>=</strong> df<strong>.</strong>name<strong>.</strong>astype(str)<br>    df<strong>.</strong>states <strong>=</strong> df<strong>.</strong>states<strong>.</strong>astype(str)<br>    <em># removing whitespaces</em><br>    df[&#39;states&#39;] <strong>=</strong> df[&#39;states&#39;]<strong>.</strong>apply(<strong>lambda</strong> x: x<strong>.</strong>strip())<br>    <em># removing quotes</em><br>    df<strong>.</strong>states <strong>=</strong> df<strong>.</strong>states<strong>.</strong>apply(<strong>lambda</strong> x: x<strong>.</strong>replace(&#39;&quot;&#39;,&quot;&quot;))<br>    df<strong>.</strong>name <strong>=</strong> df<strong>.</strong>name<strong>.</strong>apply(<strong>lambda</strong> x: x<strong>.</strong>replace(&#39;&quot;&#39;,&quot;&quot;))<br><br>    <strong>return</strong> df</pre><blockquote>We also change the types of the columns to be able to use it later.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*3GXq7PXZHKGf6emQURO_Ig.png" /><figcaption>Preview</figcaption></figure><h3>Finding Common Ground</h3><p>Now, we start by writing a function that can find all the common states.</p><ul><li>We start by creating a map of state and name</li><li>We also create a reverse index for the same</li><li>We use the collections. Counter class to get the most common count and find the state using the reverse lookup index</li><li>We find the top 5 states and print it</li></ul><pre><strong>def</strong> common(show_counter<strong>=False</strong>, show_prez<strong>=False</strong>):<br>    &quot;&quot;&quot;<br>    Find the common data points in the dataset<br>    &quot;&quot;&quot;<br>    <strong>from</strong> collections <strong>import</strong> Counter<br>    mapping_state_name <strong>=</strong> {state:name <strong>for</strong> state,name <strong>in</strong> zip(df[&#39;states&#39;]<strong>.</strong>unique(), df[&#39;name&#39;]<strong>.</strong>unique())}<br>    df[&#39;states&#39;] <strong>=</strong> df[&#39;states&#39;]<br>    counter <strong>=</strong> Counter(df[&#39;states&#39;]<strong>.</strong>values)<br>    counter_vice_versa <strong>=</strong> {counter[key]: key <strong>for</strong> key <strong>in</strong> counter<strong>.</strong>keys()}<br>    max_counter <strong>=</strong> max(counter)<br>    count <strong>=</strong> sorted(counter<strong>.</strong>values())<br>    second_largest_count <strong>=</strong> count[<strong>-</strong>2]<br>    print(f&quot;Maximum occured state: {max_counter}.&quot;)<br>    print(f&quot;Second largest state: {counter_vice_versa[second_largest_count]}&quot;)<br>    print(f&quot;Third largest state: {counter_vice_versa[count[<strong>-</strong>3]]}&quot;)<br>    print(f&quot;Fourth largest state: {counter_vice_versa[count[<strong>-</strong>4]]}&quot;)<br>    print(f&quot;Fifth largest state: {counter_vice_versa[count[<strong>-</strong>5]]}&quot;)<br>    <em># presidents with the max_counter</em><br>    <strong>if</strong> show_prez:<br>        <strong>for</strong> name, state <strong>in</strong> zip(df[&#39;name&#39;], df[&#39;states&#39;]):<br>            <strong>if</strong> state <strong>==</strong> max_counter:<br>                print(f&quot;President with {max_counter} was {name}.&quot;)<br>    <strong>if</strong> show_counter:<br>        print(counter<strong>.</strong>keys())<br>common(<strong>False</strong>, <strong>True</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*aqZAC3c_wmE5DF0L9KER0Q.png" /><figcaption>Preview of the output</figcaption></figure><h3>Another way of finding common ground using a matrix-based approach</h3><p>We also use a different technique to get the common states. We plot a heatmap to get the results in a visually appealing way.</p><pre>matrix_based <strong>=</strong> df<strong>.</strong>groupby(&quot;name&quot;)[&#39;states&#39;]<strong>.</strong>value_counts()<strong>.</strong>unstack()<strong>.</strong>fillna(0)<br>sns<strong>.</strong>heatmap(matrix_based, vmax<strong>=</strong>1, vmin<strong>=</strong>0, cmap<strong>=</strong>&quot;gray&quot;)<br>plt<strong>.</strong>title(&quot;Finding common ground&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*Ee1e8yZREOtfsmzHk4CR9A.png" /><figcaption>Preview</figcaption></figure><ul><li>Here you can see, for every president in a particular state, we have 1.0, and if a president is repeated, then we have two or 1s in a column</li><li>This is a nice representation of how the presidents are distributed statewide</li><li>This heatmap shows that values that are<strong> 0 have black color</strong> and<strong> white denotes 1.0</strong></li></ul><h3>Further analysis using data from another source</h3><p>We will be using data from the CDC for each state to find some interesting insights.</p><pre><strong>def</strong> load_data_from_cdc(save<strong>=False</strong>):<br>    &quot;&quot;&quot;<br>    Load data from CDC<br>    &quot;&quot;&quot;<br>    <em># reading the html table at index 0</em><br>    df_states_fact <strong>=</strong> pd<strong>.</strong>read_html(&quot;https://www.cdc.gov/nchs/fastats/state-and-territorial-data.htm&quot;)[0]<br>    <em># changing column (for join)</em><br>    df_states_fact[&#39;states&#39;] <strong>=</strong> df_states_fact[&#39;State/Territory&#39;]<br>    <em># removing old column</em><br>    df_states_fact<strong>.</strong>drop(columns<strong>=</strong>[&#39;State/Territory&#39;], inplace<strong>=True</strong>)<br>    merged <strong>=</strong> pd<strong>.</strong>merge(df, df_states_fact)<br>    <em># Save the csv</em><br>    <strong>if</strong> save:<br>        merged<strong>.</strong>to_csv(index<strong>=False</strong>)<br>    merged<strong>.</strong>Births <strong>=</strong> merged<strong>.</strong>Births<strong>.</strong>astype(float)<br>    merged[&#39;Fertility Rate&#39;] <strong>=</strong> merged[&#39;Fertility Rate&#39;]<strong>.</strong>astype(float)<br>    merged[&#39;Death Rate&#39;] <strong>=</strong> merged[&#39;Death Rate&#39;]<strong>.</strong>astype(float)<br>    <strong>return</strong> merged<br><br>df_merged <strong>=</strong> load_data_from_cdc()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*WbiX2QjNKiutii0EUwrvEQ.png" /><figcaption>A correlation matrix</figcaption></figure><ul><li>The Correlation matrix does not show much of a relationship between the data points.</li><li>Let’s try to change the name and states into numeric values and then try to perform correlation</li></ul><pre><strong>def</strong> apply_transformations(df: pd<strong>.</strong>DataFrame):<br>    &quot;&quot;&quot;<br>    Apply transformation on states and name<br>    &quot;&quot;&quot;<br>    <em># mapping of states and names</em><br>    mapping_states <strong>=</strong> {state: index <strong>for</strong> index, state <strong>in</strong> enumerate(df[&#39;states&#39;])}<br>    mapping_name <strong>=</strong> {name: index <strong>for</strong> index, name <strong>in</strong> enumerate(df[&#39;name&#39;])}<br><br>    df<strong>.</strong>states <strong>=</strong> df<strong>.</strong>states<strong>.</strong>apply(<strong>lambda</strong> x: mapping_states[x])<br>    df<strong>.</strong>name <strong>=</strong> df<strong>.</strong>name<strong>.</strong>apply(<strong>lambda</strong> x: mapping_name[x])<br>    <strong>return</strong> df<br><br><br>df_transformed <strong>=</strong> apply_transformations(df_merged)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*OOSYa6BKwejt1pnf2yxVNA.png" /><figcaption>Transformed dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*3NdmeLxOXucvtBmY_BNloA.png" /><figcaption>New results</figcaption></figure><h3>Some interesting plots</h3><p>Let us plot some interesting datapoints</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*uD1P2FiV317P7mbKAz1k8A.png" /><figcaption>Death rate and Births</figcaption></figure><h3>Plotting maps</h3><p>To better understand the data, let us plot a map with all the marked presidents. To solve this, we combine a latitude and longitude dataset with our US Presidents dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*wM0uDKg11x5fopzPApcc2w.png" /><figcaption>Combining dataset</figcaption></figure><pre>map_osm <strong>=</strong> folium<strong>.</strong>Map(width<strong>=</strong>500,height<strong>=</strong>500,location<strong>=</strong>[37.0902, <strong>-</strong>95.7129], zoom_start<strong>=</strong>3)<br>df_final<strong>.</strong>apply(<strong>lambda</strong> row:folium<strong>.</strong>CircleMarker(location<strong>=</strong>[row[&quot;Latitude&quot;], <br>                                                  row[&quot;Longitude&quot;]])<strong>.</strong>add_to(map_osm),<br>         axis<strong>=</strong>1)</pre><blockquote>We then plot our map using Folium</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*enBGJq5NwIL-cUaqeoLAmA.png" /><figcaption>Our Map</figcaption></figure><h3>Conclusion</h3><ul><li>On analyzing the data, we found some very fascinating insights</li><li>We tried to combine data from different sources to get our answers</li><li>We used different ways to visualize our data</li><li>We found the most common states</li><li>We could have performed better if we had access to a wide variety of data about the state, such as education, income, etc</li></ul><p>Access notebook here: <a href="https://aaditkapoor.github.io/president_analysis.html">https://aaditkapoor.github.io/president_analysis.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdd20cc22086" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/finding-common-ground-us-presidents-state-analysis-cdd20cc22086">Finding Common Ground: US Presidents State Analysis</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vice Presidential and Presidential Debate Analysis using Data Science]]></title>
            <link>https://medium.com/data-science/vice-presidential-and-presidential-debate-analysis-using-data-science-61d542f0c974?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/61d542f0c974</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Sat, 24 Oct 2020 06:17:12 GMT</pubDate>
            <atom:updated>2020-10-24T16:57:37.798Z</atom:updated>
            <content:encoded><![CDATA[<h4>Debate Analysis using Data Science: Using YouTube Comments to find the true intent of voters</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*-_-RyOvgu2Q3dBhf" /><figcaption><a href="https://unsplash.com/photos/5DDYHjk_KMU">Image Source</a></figcaption></figure><h3>Introduction</h3><blockquote>I believe Data Science allows me to express my curiosity in ways I’d never imagine. The coolest thing in Data Science is that I see data not as numbers but as an <strong>opportunity</strong> (business problem), <strong>insights</strong>(predictive modeling, stats, and data wrangling), and<strong> improvement</strong> (metrics). With this thought in mind, I decided to <strong>analyze the YouTube Comments of VP and Presidential debates.</strong></blockquote><p>After getting mixed results from the news sources, I thought to analyze the Vice Presidential and Presidential debates using Data Science.<br> The idea is to use YouTube comments as a medium to get the sentiment regarding the debate and getting insights from the data. In this analysis, we <strong>plot common phrases</strong>,<strong> common words</strong>, we also <strong>analyze sentiment</strong> and in the end for all my data science practitioners I present them a<strong> full-fledged dataset containing YouTube Comments of VP and Presidential debates</strong>.</p><h3>How and Why</h3><p>Why: After getting mixed results from the news sources about the outcome of the debate, I decided to use data science to help me see the outcome of the result. With the elections around the corner, technology or to be precise analytics plays a key role in shaping our thoughts and supporting our hypothesis.</p><p>How: To Analyze YouTube Comments we use Python and various other NLP Libraries followed by some data visualization tools. We will use the wonders of the awesome data wrangling library known as Pandas and we hope to find some interesting insights.</p><h3>Requirements</h3><p>For this project we require:</p><ul><li>Python 3.8</li><li>Pandas</li><li>Scikit-Learn</li><li>Numpy</li><li>Seaborn</li><li>NLTK</li><li>Wordcloud</li><li>TextBlob</li></ul><h3>Creation of the Dataset</h3><p>The dataset contains YouTube comments on the most popular/watched VP and Presidential debates. We use the YouTube Data API to get all comments (Due to the size limitation, we only get 100 comments per video). The videos have been selected through careful examination by the author, to be precise we focused on the highest number of views and the highest number of YouTube comments.</p><pre>def clean_text(string):<br>    string = re.sub(r&#39;[^\w\s]&#39;, &#39;&#39;, string) <br>    return &#39;&#39;.join(i for i in string if ord(i) &lt; 128)</pre><pre>def remove_stopwords(string):<br>    stop_words = set(stopwords.words(&#39;english&#39;)) <br>    word_tokens = word_tokenize(string) <br>    filtered_sentence = [w for w in word_tokens if not w in stop_words] <br>    filtered_sentence = [] <br>  <br>    for w in word_tokens: <br>        if w not in stop_words: <br>            filtered_sentence.append(w)</pre><pre>return &#39; &#39;.join(filtered_sentence)</pre><blockquote>These functions define the cleaning of the text and removing stopwords.</blockquote><h4>YouTube Comments Extraction</h4><p>The following function defines the video_ids we scrap and some basic extraction code. We get our data in the form of JSON, then we perform preprocessing and ultimately combine all comments followed by the sentiment score (using TextBlob) into a DataFrame.</p><pre>video_ids = {&quot;vp_debate&quot;:[&#39;65twkjiwD90&#39;,&#39;t_G0ia3JOVs&#39;,&#39;xXE6I3gWiMc&#39;], &quot;prez_debate&quot;:[<br>                                                                         &#39;yW8nIA33-zY&#39;,&#39;wW1lY5jFNcQ&#39;,&#39;K8Z9Kqhrh5c&#39;]}</pre><pre>video_id = &quot;&amp;videoId=&quot;<br>content = None</pre><pre>def extract_comments(resp):<br>    &quot;&quot;&quot;<br>        Get comments from the resp (json) yt comment<br>    &quot;&quot;&quot;<br>    com = []<br>    for i in resp[&#39;items&#39;]:<br>        com.append(i[&#39;snippet&#39;][&#39;topLevelComment&#39;][&#39;snippet&#39;][&#39;textOriginal&#39;].lower())<br>    return com</pre><pre>def get_videos_comments():<br>    &quot;&quot;&quot;<br>        Extract video comments and store in the arrays<br>    &quot;&quot;&quot;<br>    vp_debate_data = []<br>    prez_debate_data = []<br>    for i,j in video_ids.items():<br>        print(&quot;Getting for: &quot;,i)<br>        for id in j:<br>            video_id = &quot;&amp;videoId=&quot; + id<br>            resp = requests.get(URL+video_id)<br>            <br>            print(&quot;Length: &quot;,len(resp.content))<br>            content = resp.json()<br>            comments = extract_comments(content)<br>            if i == &quot;vp_debate&quot;:<br>                vp_debate_data.extend(comments)<br>            else:<br>                prez_debate_data.extend(comments)<br>    return vp_debate_data, prez_debate_data<br>class Comment:<br>    def __init__(self, text):<br>        self.text = remove_stopwords(clean_text(text))<br>        # set threshold as 0.5<br>        self.sentiment = 0 if TextBlob(self.text).sentiment.subjectivity &lt;= 0.5 else 1<br>    def return_comment(self):<br>        return self</pre><pre>class DataGenerator:<br>    def __init__(self, vp_debate_data, prez_debate_data):<br>        self.vp_data = {&quot;comments&quot;:[], &quot;sentiment&quot;:[]}<br>        self.prez_data = {&quot;comments&quot;:[], &quot;sentiment&quot;:[]}<br>        for i in vp_debate_data:<br>            c = Comment(i)<br>            self.vp_data[&#39;comments&#39;].append(c.text)<br>            self.vp_data[&#39;sentiment&#39;].append(c.sentiment)<br>        for i in prez_debate_data:<br>            c = Comment(i)<br>            self.prez_data[&#39;comments&#39;].append(c.text)<br>            self.prez_data[&#39;sentiment&#39;].append(c.sentiment)<br>        self.df_vp = pd.DataFrame(self.vp_data)<br>        self.df_prez = pd.DataFrame(self.prez_data)<br>    def return_df(self):<br>        print(&quot;Loaded dataframe.&quot;)<br>        return self.df_vp, self.df_prez<br>    <br>    # return corpus given the debate key<br>    # to be used for getting the commonkeywords and plotting<br>    def get_corpus(self, key=&quot;vp_debate&quot;):<br>        corpus = []<br>        if key == &quot;vp_debate&quot;:<br>            corpus = [i for i in self.vp_data[&#39;comments&#39;]]<br>        else:<br>            corpus = [i for i in self.prez_data[&#39;comments&#39;]]  <br>        <br>        return corpus</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*x68VZIHVgM9JpAhfNXYyYg.png" /><figcaption>A preview of our data</figcaption></figure><h3>Analysis</h3><p>Upon examining the data, we first lowercase the words, then remove all punctuations, and then we remove all the stopwords. We use the popular TextBlob (<a href="https://textblob.readthedocs.io/en/dev/">https://textblob.readthedocs.io/en/dev/</a>) to perform quick sentiment analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*jeBtF8deS6IitwWj1Mi74Q.png" /><figcaption>A bar plot representing the sentiment analysis of the comments.</figcaption></figure><blockquote>This data does not provide much information but if we try categorizing or grouping by specific words then we can get a better understanding.</blockquote><h3>Most Common Words</h3><p>We find the most common words in each of the debates (i.e df_vp and df_prez). We use the popular package known as NLTK to get the most common words (after preprocessing)</p><pre>def most_common(corpus):<br>    fd = nltk.FreqDist(corpus)<br>    print(fd)<br>    fd_t10=fd.most_common(10)<br>    counter = dict(fd_t10)<br>    # pd.Dataframe from dict to show bar plot<br>    df = pd.DataFrame.from_dict(counter, orient=&#39;index&#39;)<br>    names, values = zip(*fd_t10)<br>    # plot method 1<br>    fd.plot(10)<br>    # plot method two<br>    print(&quot;Bar Plot&quot;)<br>    df.plot(kind=&#39;bar&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*ascHiryahTWAb_YsOD_lnA.png" /><figcaption>Frequency distribution of words in the VP debate video</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*76wv9UAWqg2ApJ3R9_19rA.png" /><figcaption>Frequency distribution of words in the Presidential debate video</figcaption></figure><blockquote>As you can see in the Presidential debate video the most common words are ‘trump’ and ‘biden’.</blockquote><h3>WordCloud</h3><blockquote>Word clouds or tag clouds are graphical representations of word frequency that give greater prominence to words that appear more frequently in a source text. The larger the word in the visual the more common the word was in the document(s).</blockquote><pre>def plot_wordcloud(kind=&quot;vp_debate&quot;):<br>    words = get_word_list(kind)<br>    wordcloud = WordCloud(width = 800, height = 800, <br>                background_color =&#39;white&#39;, <br>                min_font_size = 10).generate(&#39; &#39;.join(words))<br>    plt.imshow(wordcloud) <br>    plt.axis(&quot;off&quot;) <br>    plt.tight_layout(pad = 0) <br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*EK6RnE2elj73KBMnIwDjIQ.png" /><figcaption>A WordCloud representation of vp_debate and prez_debate</figcaption></figure><h3>Find Keywords: A unique way to search specific words.</h3><p>This feature finds the specified keyword (most prob top 10 words). It acts on each sentence.</p><pre>def find_keywords(word, df):<br>    &quot;&quot;&quot;<br>        Find the specified keyword (most prob top 10 words)<br>        Acts on each sentence<br>        Add to the corresponding dataframe<br>        Returns: DataFrame<br>        <br>        <br>        Used inconjunction with get_most_common_words()<br>        <br>    &quot;&quot;&quot;<br>    for i in df.comments:<br>        if word in i.split():<br>            c = Counter(i.split())<br>            if c[word] &gt;= 2:<br>                print(f&#39;Word is {word} that occured {c[word]} times where sentence is {i}\n&#39;)<br><a href="http://twitter.com/interact">@interact</a>(keyword=&quot;biden&quot;, debates=[&#39;vp_debate&#39;,&#39;prez_debate&#39;])<br>def run(keyword, debates):<br>    df = None<br>    if debates == &quot;vp_debate&quot;:<br>        df = df_vp<br>    else:<br>        df = df_prez<br>    find_keywords(keyword,df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhN-MjWyqqBgJopcFQ9qcg.png" /><figcaption>A glimpse of the output. (Demo available)</figcaption></figure><h3>Conclusion</h3><p>We see that by using Data Science we were able to gather interesting insights about the data. We can certainly improve our work by using a better sentiment classifier and testing for the different hypotheses. We can also use n-grams and compare their frequency.</p><p>Try here: https://gesis.mybinder.org/binder/v2/gh/aaditkapoor/DebateAnalysis/mainDataset here: <a href="http://www.kaggle.com/dataset/43532333d82042a1287e00672b86a2c76e76ffbc4d85569715309714635172b0">http://www.kaggle.com/dataset/43532333d82042a1287e00672b86a2c76e76ffbc4d85569715309714635172b0</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61d542f0c974" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/vice-presidential-and-presidential-debate-analysis-using-data-science-61d542f0c974">Vice Presidential and Presidential Debate Analysis using Data Science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is your Facebook Data so valuable? A method to predict humans traits through Facebook Likes.]]></title>
            <link>https://medium.com/data-science/why-is-your-facebook-data-so-valuable-a-method-to-predict-humans-traits-through-facebook-likes-e1ae036e80fb?source=rss-39b97293486c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1ae036e80fb</guid>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Aadit Kapoor]]></dc:creator>
            <pubDate>Tue, 25 Aug 2020 11:49:06 GMT</pubDate>
            <atom:updated>2020-08-28T13:48:05.616Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why is your Facebook Data so valuable? A method to predict humans traits (gender, political preference, age) through Facebook Likes.</h4><p>As the election time is approaching we will see how and why our Facebook Data is so valuable to advertisers, politicians. Facebook is the world&#39;s largest social network platform with over 2.5 billion active users. It processes data at a scale never seen before, the highly sophisticated Facebook A.I algorithms curate, categorize, and predict associations between data in an almost human way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ubfJu-nG2HbAD0wr" /><figcaption><a href="https://unsplash.com/photos/HUBofEFQ6CA">https://unsplash.com/photos/HUBofEFQ6CA</a></figcaption></figure><h3>How and Why</h3><p>Why: Due to the influx of such a vast amount of data and processing power, we will explore how we can predict human traits using just a collection of Facebook Likes. To achieve our results, we will try to replicate the popular paper that analyzed data from the <a href="https://en.wikipedia.org/wiki/Facebook%E2%80%93Cambridge_Analytica_data_scandal">Cambridge Analytica Data Scandal</a>. (<a href="https://www.pnas.org/content/110/15/5802">Paper here</a>)</p><p>How: To build a predictive model, we will utilize the <a href="https://www.propublica.org/datastore/dataset/facebook-ad-categories">Facebook Ad Categories dataset</a>. Using this, we will attempt to create a user-like sparse matrix where each category corresponds with a rating. (1 represents, the user likes the content. 0 represents, the user does not like the content)</p><h3>Requirements</h3><ul><li>Python 3.8</li><li>Scikit-Learn</li><li>Pandas</li><li>Numpy</li></ul><h3>Dataset</h3><p>Like any ML/Data Mining project, we will start by analyzing and generating the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*NXaYoSPD9jJrgwVdQI54ig.png" /><figcaption>A preview of the <a href="https://www.propublica.org/datastore/dataset/facebook-ad-categories">Facebook Ad Categories dataset</a>.</figcaption></figure><p>As you can see, the dataset is very broad and because it is crowd-sourced, these entries are of real users. It is evident from the dataset that we only need the <strong>“name”</strong> column. With this as the starting point, we can generate the next part of our dataset.</p><pre>def generate_data(number_of_users=55000, cols=cols):</pre><pre>&quot;&quot;&quot;</pre><pre>Generates random data consisting of user&#39;s likes and dislikes</pre><pre>Arguments:</pre><pre>number_of_users: int</pre><pre>Returns:</pre><pre>DataFrame</pre><pre>&quot;&quot;&quot;</pre><pre>assert number_of_users &lt;= len(cols), &quot;Number of users and cols should be less or equal.&quot;</pre><pre>index = [&quot;User {}&quot;.format(i) for i in range(1, number_of_users+1)] # Number of user</pre><pre># generic categories</pre><pre>cols = cols.tolist()</pre><pre>data = {col: [] for col in cols}</pre><pre># random liking or disliking ( 1 or 0)</pre><pre>def like_or_not(): return random.randint(0,1)</pre><pre>for col in cols:</pre><pre>#print(&quot;Adding for {}&quot;.format(col))</pre><pre>for i in range(1,number_of_users+1):</pre><pre>data[col].append(like_or_not())</pre><pre>print(&quot;Data generation complete.&quot;)</pre><pre>return pd.DataFrame(data=data, index=index), index, cols</pre><p>To generate our data, we extract the “name” column and randomly assign 1 or 0 for each user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*0jQmg4PSj3gS4kKJUXomAQ.png" /><figcaption>like_or_not function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*CupF7kS1Bg3vjscujQogyw.png" /><figcaption>A preview of our generated dataset for number_of_users=5</figcaption></figure><h4>Target Variables</h4><p>To add our target variables, we will focus on “age”, “gender” and “political”. These are the variables that we want to predict using our collection of Facebook Likes for each user.</p><pre>def generate_target_variables(number_of_users=55000,target=[&#39;age&#39;, &#39;gender&#39;, &#39;political&#39;]):<br>  # for each target we will generate random data<br>  data = {&quot;age&quot;:[], &quot;gender&quot;:[], &quot;political&quot;:[]}</pre><pre># for age (age ranged: 18 to 75)<br>  # regression<br>  data[&#39;age&#39;] = [random.randint(18, 75) for x in range(1, number_of_users+1)]</pre><pre># for gender (m or f) ( 1 or 0)<br>  # classification <br>  data[&#39;gender&#39;] = [random.randint(0,1) for x in range(1, number_of_users+1)]</pre><pre># for political<br>  # classification (1 -&gt; democratic, 0 -&gt; republican)<br>  data[&#39;political&#39;] = [random.randint(0,1) for x in range(1, number_of_users+1)]</pre><pre>return data</pre><pre># adding target variables<br>data[&#39;age&#39;] = target[&#39;age&#39;]<br>data[&#39;gender&#39;] = target[&#39;gender&#39;]<br>data[&#39;political&#39;] = target[&#39;political&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*4UDc7lx2tg1-1syAc8x4Rg.png" /><figcaption>Our target variables in the dataset</figcaption></figure><h3>Dimensionality Reduction</h3><p>As our dataset contains ad categories of about 52000+ products, we will perform dimensionality reduction using SVD. (Also mentioned in the paper)</p><p>For those of you that don’t remember SVD or PCA, we will provide a brief overview of the process.</p><blockquote><a href="https://en.wikipedia.org/wiki/Dimensionality_reduction">Dimensionality reduction</a>, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension.</blockquote><p>For SVD, we essentially transform the data using the <strong>formula</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*KBPNDVasIhK-BMz8.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Singular_value_decomposition">https://en.wikipedia.org/wiki/Singular_value_decomposition</a></figcaption></figure><blockquote>To perform our dimensionality reduction, we will use the<a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html"> sklearn TruncatedSVD</a> class where n_components=100 (according to the paper.)</blockquote><pre>def dimen_reduce(values):<br>    <br>    reduced = TruncatedSVD(n_components=100)<br>    tr = reduced.fit_transform(values)<br>    return tr</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*3xrDy9VdlhK5klMTc1A0ow.png" /><figcaption>On performing dimen_reduce, we get this transformed DataFrame</figcaption></figure><pre>def generated_reduced_df(df, index, number_of_users=55000):</pre><pre>tr = dimen_reduce(df.values)</pre><pre># 100 components</pre><pre>dimen_df = pd.DataFrame(data=tr, index=index)</pre><pre>target = generate_target_variables(len(dimen_df))</pre><pre>dimen_df[&#39;age&#39;] = target[&#39;age&#39;]</pre><pre>dimen_df[&#39;gender&#39;] = target[&#39;gender&#39;]</pre><pre>dimen_df[&#39;political&#39;] = target[&#39;political&#39;]</pre><pre>return dimen_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*02HovBaA1srcbVta7NpOIg.png" /><figcaption>Our final dataset</figcaption></figure><h3>Training</h3><p>As per the paper, we train the age target variable with Linear Regression and other parameters with Logistic Regression (classification).</p><blockquote>The paper mentions a lot of parameters but due to computational constraints, we are just predicting three variables (age, gender, political).</blockquote><pre>def split_dataset(df,test_size=0.2, stype=&quot;linear&quot;):</pre><pre>features_age, labels_age = df.drop(columns=[&#39;age&#39;]).values, df[&#39;age&#39;].values</pre><pre>features_gender, labels_gender = df.drop(columns=[&#39;gender&#39;]).values, df[&#39;gender&#39;].values</pre><pre>features_political, labels_political = df.drop(columns=[&#39;political&#39;]).values, df[&#39;political&#39;].values</pre><pre>if stype == &#39;linear&#39;:</pre><pre>x_train, x_test, y_train, y_test = train_test_split(features_age, labels_age, random_state=42, test_size=test_size)</pre><pre>return x_train, x_test, y_train, y_test</pre><pre>if stype == &#39;clas_gender&#39;:</pre><pre>x_train, x_test, y_train, y_test = train_test_split(features_gender, labels_gender, random_state=42, test_size=test_size)</pre><pre>return x_train, x_test, y_train, y_test</pre><pre>if stype == &#39;clas_pol&#39;:</pre><pre>x_train, x_test, y_train, y_test = train_test_split(features_political, labels_political, random_state=42, test_size=test_size)</pre><pre>return x_train, x_test, y_train, y_test</pre><blockquote>As gender and political target variables are in the form of 1 and 0, we use LogisticRegression to train our model. We use LinearRegression to train our model with age as the target variable.</blockquote><p>According to the paper, we apply 10 fold cross-validation with 100 components as per the SVD.</p><pre># age<br>cross_val_score(linear_reg, x_train_linear, y_train_linear, cv=10)<br>cross_val_score(linear_reg, x_train_linear, y_train_linear, cv=10)</pre><pre># gender and political<br>cross_val_score(log_reg, x_train_gender, y_train_gender, cv=10)<br>cross_val_score(log_reg, x_train_pol, y_train_pol, cv=10)</pre><h3>Conclusion</h3><p>To conclude, we see how Facebook likes can influence or predict important human traits. With enough data (n=55000), we can achieve an acceptable accuracy as measured ROC. With the influx of data, we see how easy is it for organizations to target a specific set of people if they have access to Facebook data.</p><h3>References</h3><p><strong>[1]</strong> <a href="https://www.pnas.org/content/pnas/110/15/5802.full.pdf">https://www.pnas.org/content/pnas/110/15/5802.full.pdf</a></p><h3>Links</h3><p>[1] Github: <a href="https://github.com/aaditkapoor/Facebook-Likes-Model">https://github.com/aaditkapoor/Facebook-Likes-Model</a></p><p>[2]Colab:<a href="https://colab.research.google.com/drive/1k75ODwkhEnXirVDKzyyBImPqNGISqjs6?usp=sharing">https://colab.research.google.com/drive/1k75ODwkhEnXirVDKzyyBImPqNGISqjs6?usp=sharing</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1ae036e80fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-is-your-facebook-data-so-valuable-a-method-to-predict-humans-traits-through-facebook-likes-e1ae036e80fb">Why is your Facebook Data so valuable? A method to predict humans traits through Facebook Likes.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>