<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kriti Srivastava on Medium]]></title>
        <description><![CDATA[Stories by Kriti Srivastava on Medium]]></description>
        <link>https://medium.com/@kritisrivastava2801?source=rss-7ea2fe1d0bde------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7CjkdwCgv9MphnsSLECtFQ@2x.jpeg</url>
            <title>Stories by Kriti Srivastava on Medium</title>
            <link>https://medium.com/@kritisrivastava2801?source=rss-7ea2fe1d0bde------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:53:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kritisrivastava2801/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI: From the Pre-GPT Era to the GenAI Revolution]]></title>
            <link>https://medium.com/swlh/ai-from-the-pre-gpt-era-to-the-genai-revolution-db92edbb571f?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/db92edbb571f</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Wed, 10 Sep 2025 00:28:44 GMT</pubDate>
            <atom:updated>2025-09-10T00:29:27.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hv3YSj5DPURoypxAimBc7A@2x.jpeg" /><figcaption>Photo by <a href="https://www.haupes.com/">Haupes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I recently had a moment of reflection — I’ve been in the AI space for nearly a decade, long before GPT and LLMs were on everyone’s radar.</p><p>Back then, “AI” didn’t mean writing a clever prompt and getting an instant answer. It meant rolling up your sleeves and engineering solutions from the ground up.</p><h3>The Pre-GPT Era: Solving AI the Hard Way</h3><p>In those days, if you wanted to solve a problem like text classification or image analysis, you had to design and build every piece of the pipeline yourself.</p><p>•	<strong>Natural Language &amp; Topic Identification</strong></p><p>Imagine hundreds of thousands of customer feedback comments. We couldn’t just “ask GPT to summarize.” Instead, we built topic modeling systems using algorithms like LDA or word embeddings to cluster feedback into meaningful themes. It was a mix of math, probability, and constant tweaking.</p><p>•	<strong>Sentiment Analysis</strong></p><p>Understanding if customers were happy, neutral, or angry required training custom classifiers. We experimented with SVMs, logistic regression, and early deep learning models — often needing massive amounts of labeled data to reach even decent accuracy.</p><p>•	<strong>Image Recognition</strong></p><p>Before today’s plug-and-play APIs, we built CNNs (Convolutional Neural Networks) from scratch. One of my projects involved detecting product defects in images. That meant gathering image datasets, annotating defects manually, and training models that often took days to converge.</p><p>•	<strong>Marrying Multiple Models Together</strong></p><p>The real magic (and pain!) was in the orchestration. A single solution often required multiple models working together: an NLP pipeline feeding into a classifier, combined with a rules engine, stitched together into one working product.</p><p>It was challenging, but it taught me the foundations of AI: data matters, models are only as good as the context they operate in, and the outcome depends on how well you design the system end-to-end.</p><h3>The GPT Era: AI at Your Fingertips</h3><p>Fast forward to now, and we live in a very different world.</p><p>With GPT and other large language models, we can simply prompt the system:</p><p>•	“<strong>Summarize this customer feedback.”</strong></p><p>•	“<strong>Generate sentiment scores.”</strong></p><p>•	“<strong>Classify these tickets by topic.”</strong></p><p>Tasks that once required weeks of engineering effort can now be tested in minutes. This shift has democratized AI, making it accessible to non-technical teams and accelerating innovation.</p><p>But there’s a catch: relying solely on GPT can create the illusion that AI is just about writing good prompts. In reality, the fundamentals still matter: data quality, orchestration, integration, and business context.</p><h3>The Future: Agentic AI and Beyond</h3><p>The next frontier isn’t just about generating outputs. It’s about AI systems that act.</p><p>•	An Agentic AI doesn’t just tell you the supply chain is at risk. It reroutes shipments, notifies customers, and updates dashboards automatically.</p><p>•	A Digital Twin + AI agent can simulate your factory floor, detect machine failures, and reassign workloads — all without waiting for a manager to intervene.</p><p>This is the natural evolution:</p><p>•	Pre-GPT Era → Build everything from scratch.</p><p>•	GPT Era → Generate insights faster with prompts.</p><p>•	Agentic AI Future → Autonomous systems that sense, decide, and act.</p><h3>Why This Matters</h3><p>Having lived through both the pre-GPT hustle and today’s GenAI revolution, I see a clear pattern:</p><p>•	The tools change.</p><p>•	The methods evolve.</p><p>•	But the principles remain timeless.</p><p>AI is — and will always be — about:</p><p>•	Understanding the problem deeply</p><p>•	Leveraging the right data</p><p>•	Designing systems that last</p><p>The people who thrive will be those who can combine the foundations of AI with the new power of LLMs and agents.</p><p>It’s not about hype. It’s about building things that matter.</p><p>⸻</p><blockquote>Final Thoughts</blockquote><blockquote>Looking back, I’m grateful for the years of “puzzle-solving AI” — stitching together models, handling messy data, and debugging endless pipelines. It gave me the foundation to appreciate what GPT has unlocked today.</blockquote><blockquote>Looking forward, I’m equally excited for the era of Agentic AI — where AI will move from just generating answers to taking meaningful actions.</blockquote><p>We’ve come a long way. And the journey is just getting started.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db92edbb571f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/ai-from-the-pre-gpt-era-to-the-genai-revolution-db92edbb571f">AI: From the Pre-GPT Era to the GenAI Revolution</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI + Digital Twins = Living, Acting Systems]]></title>
            <link>https://kritisrivastava2801.medium.com/agentic-ai-digital-twins-living-acting-systems-c197dabdc440?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/c197dabdc440</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Thu, 04 Sep 2025 03:29:07 GMT</pubDate>
            <atom:updated>2025-09-04T03:31:11.821Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCZE0accUNGmGB6fe-v7pA@2x.jpeg" /><figcaption>Photo by Sam Moghadam on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We’ve all talked about GenAI — tools like ChatGPT that generate text, images, and ideas.</p><p>But the next big leap isn’t just generation → it’s action.</p><p>That’s where Agentic AI and Digital Twins meet.</p><p>🔹 Digital Twins are real-time virtual replicas of machines, processes, or even people. They don’t just look like the real thing — they stay alive with data from sensors, IoT devices, and enterprise systems.</p><p>🔹 Agentic AI is the “doer” — it doesn’t just give suggestions, it plans and executes tasks across systems and tools.</p><p>Now imagine the two together:</p><p>✅ Manufacturing: Your factory twin detects a machine overheating. Agentic AI instantly pauses production, reroutes workload, orders a replacement part, and updates stakeholders — all without human intervention.</p><p>✅ Supply Chain: A global twin simulates inventory, demand, and shipping. An agent autonomously finds the fastest, cheapest route, books logistics, and alerts the customer about new delivery dates.</p><p>✅ Smart Grids: A city’s energy twin predicts a surge in demand. An agent shifts supply in real-time, purchases backup power, and balances the grid.</p><p>✅ Healthcare: A patient’s twin (built from scans, history, and genetics) suggests treatment paths, orders lab tests, and schedules follow-ups — saving doctors hours of admin work.</p><p>⸻</p><p>💡 Why this matters:</p><p>Digital Twins give us the map, Agentic AI gives us the driver.</p><p>Together, they create self-optimizing, adaptive systems that:</p><p>•	Predict problems before they occur</p><p>•	Prescribe solutions instantly</p><p>•	Act autonomously with minimal friction</p><p>The future of AI isn’t just intelligence. It’s intelligence that acts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c197dabdc440" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT — new kid in the AI classroom?]]></title>
            <link>https://medium.datadriveninvestor.com/gpt-new-kid-in-the-ai-classroom-a6e23473abbd?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6e23473abbd</guid>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 19:34:28 GMT</pubDate>
            <atom:updated>2023-08-26T05:18:44.226Z</atom:updated>
            <content:encoded><![CDATA[<h3>GPT — new kid in the AI classroom?</h3><h4>This is not a technical explanation!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rPJJ-ldefVKwwTP9" /><figcaption>Photo by <a href="https://unsplash.com/@jannerboy62?utm_source=medium&amp;utm_medium=referral">Nick Fewings</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>GPT is the new buzzword that everyone is talking about these days. There has been a shift from coffee discussions around AI to Deep Learning a few years back and now about GPT. We’ve all been a part of these conversations in the last several months when we’ve heard (and/or given our feedback) about ChatGPT, PI, BARD, CoPilot, and other technologies.</p><ul><li>“It answers your questions”</li><li>“It talks to you, it is amazing”</li><li>“It writes code for you, writes articles for you, creates a painting for you”</li><li>“It is going to take away your job!”</li></ul><p>These are the top things that come to our mind when we talk about “GPT” related apps/tools/technology! What is it really? I’d cut to the chase and try to explain it in the simplest way to my knowledge.</p><h3>But what is GPT (or any of those buzzwords)?</h3><blockquote>It is segment of Artificial Intelligence (AI) that has been “trained” on massive amount of documents, words, datasets over a period of time, and has developed the ability to predict the next word in the sentence in a “smart” way.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0thPQX1t48GMCDHrdJnspQ.png" /><figcaption>This is a rather simpler explanation of where GPT is generated (Image by Author)</figcaption></figure><p><strong><em>Can you explain GPT to a 5-year-old? — </em></strong>I asked one of the many LLM-based apps this question multiple times and these are some of the responses I got!</p><ul><li><strong><em>Answer 1:</em></strong><em> “GPT is like a really fun robot that knows a lot of words and can talk to people. It can learn from all the words it sees and can use those words to have conversations with people. It’s a lot like a really smart talking dictionary.”</em></li><li><strong><em>Answer 2:</em></strong><em> “GPT is a computer program that can be given lots of text and then asked to generate text that sounds similar to the stuff it was given. It can make up new sentences, poems, stories, or even code programs — all based on the examples it was given! And the amazing thing is, it doesn’t just copy and paste — it learns from the examples and figures out the patterns that make the text sound right.”</em></li></ul><p>I liked the first response better (there is more chance of a 5-year-old knowing about a “robot” than a “computer program”!) I am sure if you ask the same question (using the exact same words in the same sequence), you’d get a different sentence explanation. Try it?</p><h3>Large Language Models(LLM) — foundation for GPTs</h3><p>ChatGPT or any of these apps are based on the concept of Large Language Models (LLM). Think of LLM as a really large machine — fed with a vast number and variety of documents— for a really long time, etc. In the figure below, we have other examples of LLM in recent years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HC-C_TRskJNtOhWd.jpeg" /><figcaption>Source: nextbigfuture.com/2023/04/timeline-of-open-and-proprietary-large-language-models.html</figcaption></figure><p>Let’s break LLM into smaller chunks with minimal technical aspects, so that we may comprehend the backbone of GPT-related technology better:</p><ul><li>LLM uses complex algorithms to analyze large datasets and learn the patterns and connections between words and phrases. It also captures long-range dependencies between words. So, as much as it can build a connection between the immediate next words, it can also build a connection between words far off in a sentence, paragraph, or document.</li><li>They are the “word guessers”. They are trying to predict what the next words would be if the same sentence were to appear on the internet.</li><li>The questions/queries asked LLM is called a “prompt” and the answer we get back is called a “response”. A trained model can be prompted to generate a response that could be newly generated text, summarized text, or sentiment analysis.</li><li>It is interesting to note that these “buzzwords” like ChatGPT, CoPilot, Claude, Bard AI, and LaMDA are all based on the Large Language Model (LLM). The difference is in their use case and the company that developed them.</li></ul><p>Keeping this as a high-level introduction to the concept, I would be glad to get your perception of the topic and how you would want to use it in your respective area!</p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p><a href="https://ddintel.datadriveninvestor.com/"><em>DDIntel</em> </a>captures the more notable pieces from our <a href="https://www.datadriveninvestor.com/"><em>main site</em></a> and our popular <a href="https://medium.datadriveninvestor.com/"><em>DDI Medium publication</em></a>. Check us out for more insightful work from our community.</p><p><a href="https://www.aitoolverse.com/register">Register </a>on AItoolverse (alpha) to get 50 DDINs</p><p>Support DDI AI Art Series: <a href="https://heartq.net/collections/ddi-ai-art-series">https://heartq.net/collections/ddi-ai-art-series</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6e23473abbd" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/gpt-new-kid-in-the-ai-classroom-a6e23473abbd">GPT — new kid in the AI classroom?</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Trees — First step towards Classification!]]></title>
            <link>https://medium.datadriveninvestor.com/decision-trees-first-step-towards-classification-b531018e3ebc?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/b531018e3ebc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[decision-tree]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Fri, 18 Feb 2022 23:30:15 GMT</pubDate>
            <atom:updated>2022-08-04T03:50:04.271Z</atom:updated>
            <content:encoded><![CDATA[<h3>Decision Trees — First step towards Classification!</h3><h4>Using Decision Trees to explain ML as a Solution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j5RUyLGLr3fYAxKy" /><figcaption>Photo by <a href="https://unsplash.com/@lukealrich?utm_source=medium&amp;utm_medium=referral">Luke Richardson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>A Decision Tree </strong>can be understood as the dataset represented in the form of an inverted tree, which would help us in making the next decision. It is called a Decision tree because the tree is made with a lot of combinations of If-Then-Else ‘decisions’ derived from the dataset.</p><p>I call this — The first step towards classification- because it is the easiest way to understand classification. Some might say that Logistic Regression is the first step towards classification, but most of our State-of-the-Art algorithms for classifications like Random Forest, XGBoost, LightGBM, etc. are tree-based models. Also, understanding a Decision tree in and out is simpler than anything else I’ve come across. Hence, Decision Trees!</p><h3>Breaking down a Decision Tree</h3><h4>1. When to use a Decision tree for Classification?</h4><p>Before starting with any classification algorithm, we need to understand if the data is even fit for classification. So below is a quick image difference that shows what kind of data will be the right fit for classification. If the data has a linear distribution (like in the first image below), it is a regression problem. If the data has a non-linear distribution, it’s a classification problem.</p><p>Although decision trees can be used for both regression and classification problems, in this blog we will be only focusing on the classification side of trees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*cX63MZ0cgYzr0MuAnZ4Oow.jpeg" /><figcaption>Image by Author</figcaption></figure><h4>2. What should the data look like?</h4><p>As I said, a decision tree looks like an inverted tree. Now, this inverted tree comes from a tabular dataset. <em>So all we need is a tabular dataset having a few feature columns (independent variables) and our target column (dependent variable)</em>.</p><p>The variables/features/columns of the dataset used for any machine learning model can be broken down into two parts — the independent and dependent variables. There can be multiple independent variables, but only one dependent variable.</p><p>The <strong>Independent variables </strong>of the dataset are the variables that are given already, and they will be used to create our machine learning models. They are the building blocks of our machine learning model.</p><p>The <strong>Dependent part </strong>of the dataset is what we are trying to predict, and its value depends on the independent variables.</p><p>For example, let&#39;s see the data snapshot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*o_KaMWSAqcdaHKgFyEtL0Q.jpeg" /><figcaption>Image by Author</figcaption></figure><p>This data, as we can see is a tabular dataset. The variables — Outlook, Wind, and Temp are independent variables. Sailing here is a dependent/target variable. <em>Usually, in the real-world scenario, you will not get your target variable in a straightforward way. You’ll have to create it or discover it based on the kind of use case. But that is for another day. Today we’ll only discuss the decision tree as a whole when everything in terms of data is provided to you.</em></p><h4>3. Structure of a Decision Tree</h4><p>A decision tree looks something like the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*G62PaycfdcLdNC72xft99w.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Understanding it is pretty simple. We have:</p><ul><li>An inverted Tree</li><li>Read in Top-down approach</li><li>Decision Nodes are created from independent variables</li><li>The topmost node is called Root Node. Intermediate nodes are called Child nodes.</li><li>The end Node is created from the dependent variable and is called the Leaf node.</li><li>The branches are called Edges which represent the decision rules</li><li>The tree starts with Root Node and ends with Leaf Node</li><li>At each new node level, a new split of data happens.</li><li>Total levels in the tree are referred to as the Depth of the Tree</li></ul><h4>4. Convert a Tabular Data to Decision Tree</h4><ul><li>The first step to convert tabular data into a tree is to create a tree for each feature-Target combination separately. In this problem, I have a dataset that has independent variables like Outlook and Wind and the values are provided for different days of observation. And I want to predict if there will be a Sailing of ship based on these two features. To keep the explanation simple, I will be using only two features right now.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*0yFTAq_OevXlWOoGh9mVDw.jpeg" /><figcaption>Dataset (Image by Author)</figcaption></figure><ul><li>First, we use only the Outlook — Sailing Combination and create a tree. Since we have three different categories in outlook, we get three different decision branches. We calculate the Sailing: No Sailing records for each category — Sunny, Rain, and Cloudy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*djTNrCmioy3kbcPeXW-WwA.jpeg" /><figcaption>Image by Author</figcaption></figure><ul><li>Next, we look into the combination of Wind-Sailing. Since we only have two categories in this feature — Yes or No, we create two branches for wind and calculate the target variable distribution for each of the categories. The tree looks something like this-</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*e9Fg6QLQBvm9MCLFBA61mA.jpeg" /><figcaption>Image by Author</figcaption></figure><h4>5. What should be on the Top?</h4><p>If our dataset had only one feature, let&#39;s say only Wind and Sailing, we could&#39;ve left our tree like this. This is going to be our final decision tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*e9Fg6QLQBvm9MCLFBA61mA.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Wind =Yes -&gt; 80% Sailing, 20% Not Sailing</p><p>Wind = No -&gt; 60% Sailing, 40% Not Sailing</p><p>So if we need to make a decision on the next record from this dataset, we will predict<strong> Sailing = Yes</strong> because, in both scenarios, we have more percentage of Sailing. However, when it’s windy, we can predict with 80% confidence that there will be Sailing, whereas if it’s not windy, we can predict with only 60% confidence that there will be Sailing.</p><p>But in an ideal scenario, you’ll have more than one independent variable. So we will have to combine the two decision trees together somehow.</p><p>The probable trees when we have Wind as the root node would be —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tGnFmgHJTbFJkkY22inqVg.jpeg" /><figcaption>Wind as the Root Node</figcaption></figure><p>and when we have Outlook as our root node would be —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*udngX6AjkjRrslI9PFR8xQ.png" /><figcaption>Outlook as Root Node</figcaption></figure><p>Now the obvious question arises — which node should be on top, or the root node? To solve this, we have mainly two different methods which can help us come to a conclusion about how the order of variables should be in the decision tree. <strong>These methods are — Gini Impurity and Information Gain. Both of these methods help in the split of data at each level in the decision tree.</strong></p><p>We will thoroughly go over these two methods in the next article, but for now, let’s just see the formula for these two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*PrSWh2Pi5SiJErc5" /><figcaption>Information Gain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*esjRwGRgVXPmw1g3nzLFjQ.png" /><figcaption>Gini Index</figcaption></figure><p><em>To summarize, we saw what a decision tree is, its structure, what kind of data can be used for trees, how to convert tabular data into a tree, and what could be some of the combinations when we have more than one independent feature in the dataset.</em></p><p>Meanwhile, you can check my previous blogs for the data preprocessing and ML project flow in the following order:</p><ol><li><a href="https://medium.com/datadriveninvestor/data-science-for-non-data-scientists-part-1-2689f422bea0">Data Science for Non-Data Scientists</a></li><li><a href="https://medium.com/datadriveninvestor/bridging-the-gap-between-business-data-science-cb3310b27741">Bridging the Gap between Business &amp; Data Science</a></li><li><a href="https://towardsdatascience.com/data-science-where-do-i-start-76693f6ae597">Data Science — Where do I start?</a></li><li><a href="https://towardsdatascience.com/whats-inside-the-data-aadbb28ba8cd">What’s inside the data!</a></li><li><a href="https://towardsdatascience.com/understand-the-patterns-in-the-data-f454a79ae51c">Understand the Patterns in the Data</a></li><li><a href="https://medium.com/swlh/what-to-keep-and-what-to-remove-74ba1b3cb04">Feature Engineering — What to Keep and What to remove?</a></li><li><a href="https://towardsdatascience.com/different-shades-of-machine-learning-f6ede94c438d">Different Shades of Machine Learning</a></li><li><a href="https://towardsdatascience.com/classification-lets-understand-the-basics-78baa6fbff48">Classification — Let’s understand the basics!</a></li></ol><p>I hope this blog helps someone understand the things they were not able to get earlier (and keep the questions coming)! :)</p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b531018e3ebc" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/decision-trees-first-step-towards-classification-b531018e3ebc">Decision Trees — First step towards Classification!</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classification — Let’s understand the basics!]]></title>
            <link>https://medium.com/data-science/classification-lets-understand-the-basics-78baa6fbff48?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/78baa6fbff48</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Thu, 21 Jan 2021 06:22:52 GMT</pubDate>
            <atom:updated>2021-01-22T06:48:41.930Z</atom:updated>
            <content:encoded><![CDATA[<h3>Classification — Let’s understand the basics</h3><h4>Machine Learning (Supervised Learning)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*83pl4l-WndzorgLi" /><figcaption>Photo by <a href="https://unsplash.com/@reiseuhu?utm_source=medium&amp;utm_medium=referral">Reiseuhu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my previous blog —<a href="https://towardsdatascience.com/different-shades-of-machine-learning-f6ede94c438d"> Shades of Machine Learning</a> — we discussed what are the two main types of machine learning algorithms. Just to brush up, we have <strong>Supervised Learning</strong> (where the target is known/ the data is labeled and the model works under this supervision) and <strong>Unsupervised Learning</strong> (where the target is not known/ the data is unlabeled so the algorithm doesn’t have any supervision).</p><p>In this blog, we’re just going to talk about Classification. We will be addressing some basic but important questions related to classification like — <em>what does classification really mean? What kind of data can we classify and what can’t? What are some of the Classification Algorithms?</em></p><h3>What does Classification really mean?</h3><p>Before starting with the classification, let’s understand the different parts of a dataset and its relation with the algorithms in general.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*n2pjjm7PHxSZO2yAxooA-g.jpeg" /><figcaption>Image by Author</figcaption></figure><p>I created a “hypothetical” dataset in the image above just to explain the theory (Eating pizza or coke or greens is totally your choice, please don’t hold me responsible for being fit/unfit! :P)</p><ol><li><strong>Dataset</strong> — Any data arranged in the form of rows and columns work for ML. The columns are divided into two types — Variables(could be one or more columns) and Target(always one column). The rows are our data points.</li><li><strong>Target / Label</strong> — is the column that we want to predict. It is our resultant column and we want to know for future data. In this dataset, it’s column “Fit/Unfit” marked in blue. Our entire supervised learning is dependent on this one column because this is what we want to know.</li><li><strong>Variables/ Features </strong>— are the columns other than the Target column. These columns help the ML model to predict the target for future data points. In this dataset, the variables are -&gt; “Eats Pizza”, “Drinks Coke”, “Eats Greens” and “Workout”.</li><li>You’d be thinking that <strong>what kind of column is “Person”</strong>. Well, when we feed the data in an ML algorithm, there are some columns that we do not use because we don’t want our model to “overfit” or learn each and every scenario, instead we want the algorithm to understand the general pattern and create a model to predict the same. We will discuss more overfitting in future blogs.</li></ol><p>So now that you’re familiar with how the datasets and algorithms relate, let’s come back to classification. As the name suggests, <strong>Classification means classifying the data on some grounds. It is a type of Supervised learning. In classification, the target column should be a Categorical column.</strong> If the target has only two categories like the one in the dataset above (Fit/Unfit), it&#39;s called a <strong>Binary Classification Problem</strong>. When there are more than 2 categories, it&#39;s a <strong>Multiclass Classification Problem</strong>. The “target” column is also called a “Class” in the Classification problem.</p><h3><em>What kind of data we can classify and what can’t?</em></h3><p>To classify something, we need finite categories. So we need to have a dataset with a Class label like — [0,1], [Pass, Fail], [Fit, Unfit], [Good, Better, Best], etc. If we have such type of values in the target column, we can use the Classification method to solve the problem. But if we have continuous numeric values in the target column like [100,23,44,46,44.7,24.8, … etc.], we cannot classify on such a dataset. In this case, we either convert this value into a class value, for example — {values &gt;50 will be considered 1, and &lt;50 is 0}, or we use other methods like regression to solve the problem which is out of the scope of this blog.</p><p>To understand this better, let’s take an example of the employee salary and other features dataset (below):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*4EtvhWXzAnRjiDLCx9oOpw.jpeg" /><figcaption>Data For Non-Classification Problem(Image By Author)</figcaption></figure><p>The above image consists of some data points, with Salary as our target variable. Now since Salary is a continuous number column (since salary is always numeric), we cannot treat this as a classification problem. But if we really want to treat this as a classification problem due to some reason, we can bin the target column into two categories <strong>for example — Salary&gt;70,000 as High and Salary&lt;70,000 as Low.</strong> After doing this, the dataset will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*WFj5L_94IkeoPbOaYRnTuQ.jpeg" /><figcaption>Converting Regression to Classification Problem (Image by Author)</figcaption></figure><p>So now our data is ready to be treated as a Classification Problem. Of course, there are other data processing stuff which we need to do based on the different type of algorithms used, but we at least have a categorical target now to classify on.</p><h3><em>How does an ML Algorithm or a model classify data?</em></h3><p>Now that we know how the datasets can be related to ML models, and what datasets can be used as a Classification Problem, we should also know how to solve them, right?</p><p>There are a lot of algorithms in the world that are being used to solve classification problems, and even newer algorithms are being introduced every day. Some of these algorithms are — <strong>Decision Trees,</strong> <strong>Support Vector Machines(SVM), Random Forest, Gradient Boosted Trees(GBT), K-nearest neighbors (KNN), </strong>etc. The Decision Tree is the basic algorithm to understand the classification by everyone and also to understand how the trees work in machine learning. But I like to keep my blogs short, and Decision trees deserve an entire blog, so will stop here and will try to bring up my next blog on Decision trees soon! ❤</p><p>Meanwhile, you can check my previous blogs for the data preprocessing and ML project flow in the following order:</p><ol><li><a href="https://medium.com/datadriveninvestor/data-science-for-non-data-scientists-part-1-2689f422bea0">Data Science for Non-Data Scientists</a></li><li><a href="https://medium.com/datadriveninvestor/bridging-the-gap-between-business-data-science-cb3310b27741">Bridging the Gap between Business &amp; Data Science</a></li><li><a href="https://towardsdatascience.com/data-science-where-do-i-start-76693f6ae597">Data Science — Where do I start?</a></li><li><a href="https://towardsdatascience.com/whats-inside-the-data-aadbb28ba8cd">What’s inside the data!</a></li><li><a href="https://towardsdatascience.com/understand-the-patterns-in-the-data-f454a79ae51c">Understand the Patterns in the Data</a></li><li><a href="https://medium.com/swlh/what-to-keep-and-what-to-remove-74ba1b3cb04">Feature Engineering — What to Keep and What to remove?</a></li><li><a href="https://towardsdatascience.com/different-shades-of-machine-learning-f6ede94c438d">Different Shades of Machine Learning</a></li></ol><p>I hope this blog helps someone understand the things they were not able to get earlier (and keep them questions coming)! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78baa6fbff48" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/classification-lets-understand-the-basics-78baa6fbff48">Classification — Let’s understand the basics!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Different Shades of Machine Learning]]></title>
            <link>https://medium.com/data-science/different-shades-of-machine-learning-f6ede94c438d?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6ede94c438d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Wed, 25 Nov 2020 02:01:14 GMT</pubDate>
            <atom:updated>2021-01-19T07:56:24.991Z</atom:updated>
            <content:encoded><![CDATA[<h3>Shades of Machine Learning</h3><h4>Supervised vs Unsupervised Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7D0Eot52o7doM4NA" /><figcaption>Photo by <a href="https://unsplash.com/@inakihxz?utm_source=medium&amp;utm_medium=referral">Iñaki del Olmo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the last few blogs, we discussed various methods of cleaning and transforming data at different scales before applying any ML algorithm to it (you can find the links at the bottom of this article). This preprocessing of the data is required because dirty/messy data would not make any sense to the model and to us. So now that we have learned how to clean the data, let’s look at the main picture — <strong>the different ML techniques to solve the different types of problems around us.</strong></p><p>Let’s start with the basic ML Techniques in the world!</p><h3>Supervised vs Unsupervised Learning</h3><h4>Supervised Learning</h4><p>As the name suggests,<strong> <em>supervised learning is learning under some supervision</em></strong><em>.</em> For example, what you learn in school is supervised learning because there are books and teachers who supervise you and guide you towards the end goal. <strong>Similarly in terms of machine learning, when the model is able to learn the “if this — then this” pattern, it is called supervised learning.</strong></p><p>Supervised learning consists of Classification and Regression techniques because in both cases, the end goal is set. In the case of classification, one of the different classes/categories/targets are supposed to be predicted from a finite list. In regression, a continuous number is supposed to be predicted. But in both cases, historical data tells what should be predicted when a particular sequence of things happens.</p><p><strong><em>For example, </em></strong><em>you want to decide if today you should go “out in the park” or “sit at home”? The first thing which will come to your mind is the weather — if it rains, you would “sit at home”, if it’s sunny, you’d definitely want to go out in the park because that’s the logical thing to do (unless you are a couch potato and just want to sit inside anyway and finish that series!).</em></p><p><em>But if you ask a computer the same question, it doesn’t understand human logic. So it needs some kind of supervision or a set of rules to give an answer, which is what the historical data will do. The historical data will help the computer understand patterns like, in the past, you’ve stayed indoors 8/10 times when it rains. So it’ll develop a rule in its memory that — when it rains -&gt; stay at home. And the next time you ask the computer this question, it’ll ask you to stay indoors when it rains.</em></p><p>Easy right? This is how we learned daily stuff as kids by remembering what our adults told us, or just watching what they did. Supervised learning is pretty friendly in that matter. It wants us to succeed, and it is ready to help us in whatever way possible. We just need to find the right datasets.</p><h4>Unsupervised Learning</h4><p>Unsupervised Learning is not so friendly!<strong> It doesn’t supervise the model in the ML journey, </strong>but that’s the thing, unsupervised learning helps you get more and more curious. It’ll help you explore patterns and groups you would’ve not thought about before. The data is not labeled before, the target is not known for unsupervised learning data.</p><p><strong><em>For example,</em></strong><em> you have to divide the people in your office into different groups for Holiday Activity, and instead of randomly dividing everyone into groups, you come up with some features like height, weight, department, designation, gender, the floor they sit on, team, etc. And based on these features, you make several groups. These different groups are called clusters, and the features of people in each cluster would be different from those of the other cluster’s people, but very similar to the people in the same cluster. So in this case, instead of coming up with the number of groups beforehand, we let the distinguishing features of people decide the group they want to go in or create a new group if their features don’t exist in any previous cluster.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*rPkbJeLSY45YzP9gTeQ9IA.jpeg" /><figcaption>Image from <a href="https://www.researchgate.net/figure/Supervised-VS-Unsupervised-Learning_fig6_323108787">Introduction to Machine Learning</a></figcaption></figure><p>Now that we have understood the difference between supervised and unsupervised learning, the image below explains some basic ML techniques in these categories and their distinct features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*pxn2DiE-x_iyn_hUG1mM9g@2x.jpeg" /><figcaption>Image by Author</figcaption></figure><p>In the next blog, we will try to understand some techniques of supervised learning in-depth. Meanwhile, you can check my previous blogs for the data preprocessing and ML project flow:</p><ol><li><a href="https://medium.com/datadriveninvestor/data-science-for-non-data-scientists-part-1-2689f422bea0">Data Science for Non-Data Scientists</a></li><li><a href="https://medium.com/datadriveninvestor/bridging-the-gap-between-business-data-science-cb3310b27741">Bridging the Gap between Business &amp; Data Science</a></li><li><a href="https://towardsdatascience.com/data-science-where-do-i-start-76693f6ae597">Data Science — Where do I start?</a></li><li><a href="https://towardsdatascience.com/whats-inside-the-data-aadbb28ba8cd">What’s inside the data!</a></li><li><a href="https://towardsdatascience.com/understand-the-patterns-in-the-data-f454a79ae51c">Understand the Patterns in the Data</a></li><li><a href="https://medium.com/swlh/what-to-keep-and-what-to-remove-74ba1b3cb04">Feature Engineering — What to Keep and What to remove?</a></li></ol><p>I hope this blog helps someone understand the things they were not able to get earlier (and keep them questions coming)! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6ede94c438d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/different-shades-of-machine-learning-f6ede94c438d">Different Shades of Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What to Keep and What to Remove]]></title>
            <link>https://medium.com/swlh/what-to-keep-and-what-to-remove-74ba1b3cb04?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/74ba1b3cb04</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Wed, 19 Aug 2020 06:16:08 GMT</pubDate>
            <atom:updated>2020-08-22T18:50:15.533Z</atom:updated>
            <content:encoded><![CDATA[<h3>Feature Engineering — What to Keep and What to Remove</h3><h4>Performing feature engineering on the dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XJcXE-ZfTAe0NUhN" /><figcaption>Photo by <a href="https://unsplash.com/@gndclouds?utm_source=medium&amp;utm_medium=referral">William Felker</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The next step after exploring the patterns in data is feature engineering. Any operation performed on the features/columns which could help us in making a prediction from the data could be termed as Feature Engineering. This would include the following at high-level:</p><ol><li>adding new features</li><li>eliminating some of the features which tell the same story</li><li>combining several features together</li><li>breaking down a feature into multiple features</li></ol><h4>Adding new features</h4><p>Suppose you want to predict sales of ice-cream or gloves, or umbrella. What is common in these items? The sales of all these items are dependent on “weather” and “location”. Ice-creams sell more during summer or hotter areas, gloves are sold more in colder weather (winter) or colder regions, and we definitely need an umbrella when there’s rain. So if you have the historical sales data for all these items, what would help your model to learn the patterns more would be to add the weather and the selling areas at each data level.</p><h4>Eliminating some of the features which tell the same story</h4><p>For explanation purpose, I made up a sample dataset which has data of different phone brands, something like the one below. Let us analyze this data and figure out why we should remove/eliminate some columns-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*vmiMFvhSoOs-cNknPFwVrg.jpeg" /><figcaption>Image by Author</figcaption></figure><ol><li>Now in this dataset, if we look carefully, there is a column for the <strong>brand</strong> name, a column for the <strong>model</strong> name, and there’s another column which says <strong>Phone</strong> (which basically contains both brand and model name). So if we see this situation, we don’t need the column Phone because the data in this column is already present in other columns, and split data is better than the aggregated data in this case.</li><li>There is another column that is not adding any value to the dataset — <strong>Memory scale.</strong> All the memory values are in terms of “GB”, hence there is no need to keep an additional column that fails to show any variation in the dataset, because it’s not going to help our model learn different patterns.</li></ol><h4>Combining several features to create new features</h4><p>This means we can use 2–3 features or rows and create a new feature that explains the data better. For example, in the above dataset, some of the features which we can create could be — count of phones in each brand, % share of each phone in respective brand, count of phones available in different memory size, price per unit memory, etc. This will help the model understand the data at a granular level.</p><h4>Breaking down a feature into multiple features</h4><p>The most common example in this segment is Date and Address. A date mostly consists of Year, Month, and Day, let’s say in the form of ‘07/28/2019’. So if we break down the Date column into 2019, 7 or July, and 28, it’ll help us join the tables to various other tables in an easier way, and also will be easy to manipulate the data, because now instead of a date format, we have to deal with numbers which are a lot easier.</p><p>For the same easier data manipulation and easier data joins reason, we break down the Address data (721 Main St., Apt 24, Dallas, TX-75432) into — Street name (721 Main St.), Apartment number/ House number (Apt 24), City (Dallas), State (TX/Texas), zip code (75432).</p><p>Now that we know what feature engineering is, let’s go through some of the techniques by which we can do feature engineering. There are various methods out there for feature engineering, but I will discuss some of the most common techniques &amp; practices that I use in my regular problems.</p><ol><li><strong>Lags</strong> — this means creating columns for previous timestamp records (sales 1-day back, sales 1-month back, etc. based on the use-case). This feature will help us understand, for example, what was the iPhone sale 1 day back, 2 days back, etc. This is important because most of the machine learning algorithms look at the data row-wise, and unless we don’t have the previous days&#39; records in the same row, the model will not be able to create patterns between current and previous date records efficiently.</li><li><strong>Count of categories</strong> — this could be anything as simple as count of phones in each brand, count of people buying iPhone 11pro, count of the different age groups of people buying Samsung Galaxy vs iPhone.</li><li><strong>Sum/ Mean/ Median/Cumulative sum/ Aggregate sum </strong>— of any numeric features like salary, sales, profit, age, weight, etc.</li><li><strong>Categorical Transformation Techniques</strong> (replacing values, one-hot encoding, label encoding, etc) — These techniques are used to convert the categorical features to respective numerical encoded values, because some of the algorithms (like xgboost) do not identify categorical features. The correct technique depends on the number of categories in each column, the number of categorical columns, etc. To learn more about different techniques, <a href="https://pbpython.com/categorical-encoding.html">check this blog</a> and <a href="https://www.datacamp.com/community/tutorials/categorical-data">this blog</a>.</li><li><strong>Standardization/ Normalization techniques</strong> (min-max, standard scaler, etc) — There could be some datasets where you have numerical features but they’re present at different scales (kg, $, inch, sq.ft., etc.). So for some of the machine learning methods like clustering, it is important that we have all the numbers at one scale (we will discuss about clustering more in later blogs, but for now understand it as creating groups of data points in space based on the similarity). To know more about this section, check out these blogs — <a href="https://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/">Feature Scaling Analytics Vidhya</a>, <a href="https://www.oreilly.com/library/view/machine-learning-with/9781491989371/ch04.html">Handling Numerical Data O&#39;Reilly</a>, <a href="https://machinelearningmastery.com/standardscaler-and-minmaxscaler-transforms-in-python/">Standard Scaler/MinMax Scaler.</a></li></ol><blockquote>These are some of the very general methods of creating new features, but most of the feature engineering largely depends on brainstorming on the dataset in the picture. For example, if we have a dataset for employees, vs if we have the dataset of the general transactions, feature engineering will be done in different ways.</blockquote><p>We can create these columns using various pandas functions manually. Besides these, there is a package called <a href="https://github.com/FeatureLabs/featuretools">FeatureTools</a>, which can also be explored to create new columns by combining the datasets at different levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SNelH4nRenvO75qTsvXmxg.jpeg" /><figcaption>Image by Author</figcaption></figure><p>This brings us to a (somewhat)<strong> end of Data Preprocessing Stages.</strong> Once we have the data preprocessed, we need to start looking into different ML techniques for our problem statement. We will be discussing those in the upcoming blogs. Hope y’all found this blog interesting and useful! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74ba1b3cb04" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/what-to-keep-and-what-to-remove-74ba1b3cb04">What to Keep and What to Remove</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand the Patterns in the Data]]></title>
            <link>https://medium.com/data-science/understand-the-patterns-in-the-data-f454a79ae51c?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/f454a79ae51c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[data-distribution]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Fri, 17 Jul 2020 09:52:53 GMT</pubDate>
            <atom:updated>2020-07-17T17:07:36.721Z</atom:updated>
            <content:encoded><![CDATA[<h4>The next step after importing the data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9WQ304mOPIiA_8N_" /><figcaption>Photo by <a href="https://unsplash.com/@firmbee?utm_source=medium&amp;utm_medium=referral">William Iven</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my <a href="https://towardsdatascience.com/whats-inside-the-data-aadbb28ba8cd">previous blog</a>, I explained how to clean the data, perform EDA (exploratory data analysis) and what is a basic feature engineering in brief.</p><p>So let’s say, you did a “read_csv” and imported the data. Now, what next? The next important thing to talk about is how can we understand and analyze various patterns in our dataset. This will help us address a few things :</p><p><strong>a.</strong> It will help us understand how many <strong>missing values/duplicate values</strong> are currently present in the data (which are a problem for our model in the future) — <a href="https://towardsdatascience.com/whats-inside-the-data-aadbb28ba8cd"><em>explained in the previous post</em></a></p><p><strong>b. </strong>Identify the <strong>independent</strong> (Features/attributes) and <strong>dependent</strong> variables (Target)? If there is no Target (in most of the real-world cases), understand how to create one. This will be based on the problem statement which you’re working on, and I will discuss this with some examples in later blogs.</p><p><strong>c.</strong> Understand the <strong>general distribution</strong> of each feature/attribute with respect to our Target column, and with respect to each other.</p><p><strong>d. </strong>Are there<strong> features that can be removed</strong> because they ultimately explain the same thing! (Will address this in the next blog)</p><figure><img alt="EDA Flow" src="https://cdn-images-1.medium.com/max/1024/1*1d44Ui9NmGIBO7KUzfbnnw.jpeg" /><figcaption>Source: Author</figcaption></figure><h3>Data Distribution Analysis</h3><p>To understand the Data Distribution and relationships, there are a lot of python libraries (seaborn, plotly, matplotlib, sweetviz, etc.), which will make your work easier. Some of the things to keep in mind at this stage are:</p><ol><li>Identify your numerical &amp; categorical variables. If you think a column of zipcode is numerical because it has numbers, you might want to go back and read about the categorical and numerical attributes in detail! For this article’s purpose, <strong>categorical features are those which would include a limited number of categories, whereas numerical attributes are those which contain a continuous real number</strong>. I found<a href="https://www.scribbr.com/methodology/types-of-variables/"> this link</a> very useful for the same! This part looks small, but it plays an important role later in the machine learning modeling.</li><li>Once you’ve successfully identified the different types of variables, it’s time to see their patterns and distributions. For different types of variables, you would want to use different charts/ explanations to understand them better.</li><li><strong>Categorical Features </strong>— some of the basic stats which would be helpful here are — count the unique values in each feature, the frequency distribution in each feature, how is each category related to the target column, etc. You can use a lot of graphs here, I use pie chart (for lesser categories), a bar graph for more categories, or sometimes just a normal table to do quick analysis. It totally depends on what makes the distribution easier for you to understand!</li><li><strong>Numerical Features </strong>—For the numerical attributes like salary, age, temperature, etc., create a histogram for them to understand their distribution. My go-to line of code is df[‘&lt;column&gt;’].<a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html">describe()</a>. This lets me quickly analyze the min-max-mean, etc of an attribute, without investing a lot of time in coding. There are a lot of graphs present out there which you can use to analyze the numerical features too!</li><li>To <strong>analyze the Target column</strong>, see the distribution of the data — bar graph/pie chart for 1s &amp; 0s in a classification problem, or a histogram/ scatter plot for the regression problem(continuous number).</li></ol><p>One thing to keep in mind is, that EDA (and whole data science) is very intuitive.</p><blockquote>You might think the process looks like import data -&gt; EDA -&gt; feature engineering -&gt; modeling -&gt; results, but in reality, <strong>the process is a loop </strong>and everything is connected to each other.</blockquote><p>So even if you reach the modeling stage (after crossing all the hurdles in the data preprocessing stage), you’ll have to come back to analyze more patterns, create/remove more features, and do the modeling again! Patience is the key :)</p><blockquote>There are a lot of libraries out there providing a lot of “fancy” interactive graphs, but along with a nice-looking graph, you should select the one which helps you understand the data well, and is easy to explain to others as well.</blockquote><p>Now that you’re at the end of the article, thank you for reading! If there’s anything you want me to add/change/address, please leave a response! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f454a79ae51c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/understand-the-patterns-in-the-data-f454a79ae51c">Understand the Patterns in the Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s inside the Data!]]></title>
            <link>https://medium.com/data-science/whats-inside-the-data-aadbb28ba8cd?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/aadbb28ba8cd</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Mon, 22 Jun 2020 03:05:34 GMT</pubDate>
            <atom:updated>2020-06-22T05:41:26.900Z</atom:updated>
            <content:encoded><![CDATA[<h4>Preprocessing, EDA &amp; Feature Engineering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W-QdwD4xSXWUVFe7" /><figcaption>Photo by <a href="https://unsplash.com/@franki?utm_source=medium&amp;utm_medium=referral">Franki Chamaki</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Believe it or not, this part covers about 60–70% of the entire ML work! I will not talk about the codes here, because I believe that if you know the logic, finding a code isn’t super difficult, only getting there would vary from person to person. Some places to get the codes are StackOverflow, Kaggle, Github, or anything on Google which helps in making that logic work!</p><p>Instead, let’s talk about general logic/concepts which we as data scientists keep in mind when doing the initial analysis. This would include <strong>tidying the data, understanding different patterns behind, etc</strong>.</p><p>You can go through my<a href="https://towardsdatascience.com/data-science-where-do-i-start-76693f6ae597"> previous post </a>to understand the <strong>“Where do I start with ML”</strong>, for beginners or anyone from different backgrounds who want to learn ML.</p><h3>Tidy the data</h3><blockquote><strong>Data cleansing</strong> or <strong>data cleaning</strong> is the process of detecting and correcting (or removing) corrupt or inaccurate <a href="https://en.wikipedia.org/wiki/Storage_record">records</a> from a record set, <a href="https://en.wikipedia.org/wiki/Table_(database)">table</a>, or <a href="https://en.wikipedia.org/wiki/Database">database</a> and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the <a href="https://en.wikipedia.org/wiki/Dirty_data">dirty</a> or coarse data. — (Source — Wikipedia).</blockquote><p>Why do we need to clean the data? Let’s look at the example below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*El472ul0oz6AbTe0Yluouw.jpeg" /><figcaption>Source: <a href="https://quantdare.com/author/psanchezcri/">Pablo Sanchez</a> via <a href="https://quantdare.com/data-cleansing-and-transformation/">QuantDare</a></figcaption></figure><p>There are a number of problems in this small example of “dirty data”.</p><ol><li>The ID column(which is supposed to be the primary key) should have unique values, but there are <strong>duplicate records</strong>.</li><li>Date column has<strong> different date formats</strong>, which makes it difficult to analyze later at one level.</li><li>There are <strong>missing records</strong> in some of the columns</li><li>The same name has different spellings or <strong>typo errors.</strong></li></ol><p>These are just a few examples of how the data can be dirty. There can be other things also which might mess up the data and should be removed. So cleaning the data is a very important part before analyzing the data.</p><p>Now, you know the data is messed up, so how do you fix it?</p><ol><li><strong>Best Case →Talk to the source</strong> if they can help you clean up the data.</li><li>If that is not possible, then you have to do it yourself. <strong>Different types of errors in data can be fixed in multiple ways</strong> based on the use case and dataset type.</li><li><strong>Duplicate rows</strong> can be aggregated at some levels(for example here at a student level, or city-level, or teacher level), to create unique records.</li><li><strong>Different format </strong>dates, numbers, revenues, scores, etc. should be brought to the same format, whichever is applicable.</li><li><strong>Typo errors</strong> are just unavoidable if there are humans filling up the form which is the data source. But automating the system for such data inputs can help in the long run to better capture the data.</li><li><strong>Missing records</strong> — there are multiple ways to fill that based on the column which has missing values and use cases of the project.</li></ol><p>a. You can fill using a common value (like 0 or 9999 or ‘:&lt;blanks&gt;”),</p><p>b. using the mean/median of the whole column</p><p>c. using advanced methods like KNN (k-nearest neighbors) or,</p><p>d. just leave it as is if it doesn’t make a lot of difference.</p><p>Once all the different datasets are clean, you need to <strong>bring them together.</strong> Because for Machine Learning, if we have one big file, it’s easier to feed into the model.</p><p>Once the data has been cleaned to an extent, we can move to the next step.</p><h3>Understand the Patterns Behind</h3><p>Once we have cleaned the data, we can go ahead and start analyzing it. Analysis is any useful findings/insights which can be gathered from the data. For example, if you can put your entire life in data (from the first day you were born to what you had for lunch today), and we pick a random sample from it, some of the insights which you can gain could be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*lg4TEvNYDJsrSVpUOnwdWg.jpeg" /><figcaption>Source: Author</figcaption></figure><ol><li>How many burgers did you eat this year? Out of those, how many did you have with fries on the side?</li><li>How many times did you have a meal with a friend?</li><li>How many times you had a burger when you were with your friends? And things like these.</li></ol><p>Why is the analysis necessary? It helps us understand <strong>“do we have enough samples/features to solve the end problem”</strong>?</p><p>We can understand different patterns in the data, or how multiple columns are related to each other, or the findings at different levels — like in this example, we can bring the data at a day/month level, type of food, the vehicle used, etc. It can also help us remove the columns which are not necessary for our finding. I will try to explain this step in detail in my next blog, along with codes and samples.</p><h3>Feature Engineering</h3><blockquote>Feature == Attribute == Column == Dimension</blockquote><p>Feature Engineering means, to manipulate/change the features to create new features. Why is this important? Because it helps us bring the different properties of data out.</p><p>For example- Address contains: 221b Baker Street, London. We can <strong>dissolve this column to bring out more value</strong> — City, Street, Apt#. Similarly, Birthday “Jan-21–1994” can be dissolved into Date, Month, Year, Day of the year, Week of the year, etc. These new columns make more sense than the original features.</p><p>Another example, if the whole dataset is of London, we can remove the newly created City column, because it’s<strong> not going to add any additional value. </strong>Similarly, if we have an Age column, and which is adding the same value as having the Date of Birth column, we can remove one of these.</p><p>Both the above methods are examples of <strong>Feature Engineering</strong>. There are other ways to create new features, you can add the lags, do an aggregate/cumulative sum of revenue, mean/median of columns, count of different categories, etc. So this way, we can create a lot of features that could help the model understand and predict the data well.</p><p>But, there is a thing which I came across during my Master’s- <strong>“the curse of dimensionality”.</strong> This implies that having a lot of dimensions is actually not good. Then what should we do? Well, once a simple thing to do is:<br>a. The columns which are not adding any value to the data should be removed.<br>b. More than one column implying the same detail should be removed.</p><p>Besides these, we have other methods that can automatically select the best columns for models — like feature importance, SHAP, etc. but we’ll talk about them once we’re in the stage of creating models! :) Till then, enjoy Preprocessing the data and let me know your feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aadbb28ba8cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/whats-inside-the-data-aadbb28ba8cd">What’s inside the Data!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science — Where do I start?]]></title>
            <link>https://medium.com/data-science/data-science-where-do-i-start-76693f6ae597?source=rss-7ea2fe1d0bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/76693f6ae597</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Kriti Srivastava]]></dc:creator>
            <pubDate>Sat, 13 Jun 2020 18:31:22 GMT</pubDate>
            <atom:updated>2020-06-15T15:44:52.989Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Science — Where do I start?</h3><h4>A practical starter kit to data science for non-programmers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNlLxWQa_-Z2AcHhxEQiCA.jpeg" /><figcaption>Image by Author</figcaption></figure><p>So I come across a lot of people from different backgrounds (Electronics, Mechanical, Undergraduates (1st/2nd year), Sales, Finance, etc. who want to explore the Data Science field, and some of the questions they have are these, which even I was wondering about when I started. —</p><ul><li>Can I enter in the data science world?</li><li>Can I do without coding?</li><li>Where to get the data from?</li><li>Or more importantly, where is the problem that I can solve?</li></ul><p>Well, the answer to the first question — <strong>Yes</strong>. Anyone can enter the field. It’s all about consistent learning and applying logic.</p><p>And for other questions, keep reading —</p><ul><li><strong>You definitely need coding skills.</strong> The ML coding level is not as high as what you need to become a software engineer (at least in the beginning). Still coding is an integral part of ML, because no matter how many tools you use in the starting, like Weka or Alteryx (which has drag-drop ML), ultimately <strong>you need R or Python in the long run. And SQL is important,</strong> never underestimate it!</li><li>There are various learning websites that <strong>offer free courses </strong>on these languages/concepts, like — <a href="https://medium.com/u/b32aa0132f1b">Udemy</a>, <a href="https://medium.com/u/2929690a28fb">Udacity</a>, <a href="https://medium.com/u/e18542fdcc02">DataCamp Team</a>, etc. These websites have different tracks for beginner, intermediate, advanced-level courses for Python and R needed for data science, which are very useful. And if you pay, you even get a certificate which you can post on Linkedin. There are a lot of <strong>Youtube Channels</strong> also which are <strong>dedicated to data science track for beginners,</strong> and are helpful.</li><li>So the best part of the above courses/channels is, they mostly provide the dataset that you can explore side by side. Besides these, there are a lot of <strong>open data sources </strong>these days and you can just download the data from there and start working. There are websites that provide a list of the open data sources, and you can select from them: <a href="https://www.freecodecamp.org/news/https-medium-freecodecamp-org-best-free-open-data-sources-anyone-can-use-a65b514b0f2d/"><strong>Freecodecamp</strong></a><strong>, </strong><a href="https://www.forbes.com/sites/bernardmarr/2016/02/12/big-data-35-brilliant-and-free-data-sources-for-2016/#26339872b54d"><strong>Forbes</strong></a><strong>, </strong><a href="https://learn.g2.com/open-data-sources"><strong>Learning Hub</strong></a><strong>, etc.</strong></li><li>The next best thing that happened to the world is<strong> </strong><a href="https://www.kaggle.com/"><strong>Kaggle</strong></a>! For anyone who doesn’t know (even I didn’t know 2 years back until one of my favorite professor gave us a project on this), <em>Kaggle is an online community of data scientists and machine learning practitioners, where the real organizations post their problem statement, datasets, and what they want, etc.</em> All you need to do is create an account (for free!!) and start working. It is highly recommended in the Data Science field.<strong> Anyone can participate, download, share analysis and codes, and basically anything you want to start in Machine Learning, Deep Learning, or any related field.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCbHvpi-1JGTMYTgAXr10w.jpeg" /><figcaption>Source: <a href="https://www.kaggle.com/">Kaggle.com</a></figcaption></figure><ul><li>In addition to these, there are several boot camps and dedicated courses that are offered by learning sites and provide concepts, data, and problems. They are helpful in getting visibility sometimes.</li><li>After you’ve practiced the process using the open datasets, notebooks, etc. you’ll feel more confident in touching the real data from your domain. And what will be an advantage for you is you’ll have the domain knowledge besides the ML stuff, which is the basis to solve the problems.</li><li>All the above points are about the process. To understand the ML concepts, and new trends in the field, the hacks of data manipulation, etc. some of these websites are helpful: <a href="http://www.kdnuggets.com/">KDNuggets</a>, <a href="https://towardsdatascience.com/">Towards Data Science</a>, <a href="https://machinelearningmastery.com/">Machine Learning Mastery</a>, <a href="http://www.analyticsvidhya.com/">Analytics Vidhya</a>, <a href="http://www.datasciencecentral.com/">Data Science Central</a>, etc.</li></ul><p>Okay, guess we’ve reached the end of it. You know where to learn the concepts from, practice the coding from, where to get the dataset and the problem statements, and even the codes! So what are you waiting for? Go get started! :D</p><p><em>If there are other things that can also add value, leave a response! :)</em></p><p>Also, if you want to know all the steps of ML in one place, go through my previous story <a href="https://medium.com/datadriveninvestor/data-science-for-non-data-scientists-part-1-2689f422bea0">Data Science for Non-Data Scientists (Part 1)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76693f6ae597" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-science-where-do-i-start-76693f6ae597">Data Science — Where do I start?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>