<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mudita Garg on Medium]]></title>
        <description><![CDATA[Stories by Mudita Garg on Medium]]></description>
        <link>https://medium.com/@muditagarg12?source=rss-bddec522776a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MYeSq8kWeiQ_JuJPHIoVFQ.jpeg</url>
            <title>Stories by Mudita Garg on Medium</title>
            <link>https://medium.com/@muditagarg12?source=rss-bddec522776a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:07:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@muditagarg12/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data in Production, Part 1: Building for Production (Not Just for Your Laptop)]]></title>
            <link>https://medium.com/@muditagarg12/data-in-production-part-1-building-for-production-not-just-for-your-laptop-44e43e1eca59?source=rss-bddec522776a------2</link>
            <guid isPermaLink="false">https://medium.com/p/44e43e1eca59</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Mudita Garg]]></dc:creator>
            <pubDate>Sun, 19 Apr 2026 21:15:14 GMT</pubDate>
            <atom:updated>2026-04-19T21:15:14.490Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 1 of a 3-part series on what nobody teaches you about taking data from your laptop to production.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONc8tfb2xrOURSWK9TCrBg.jpeg" /></figure><p>Has this ever happened to you? Your code looks great in development. Tests pass. Output looks exactly right. You deploy it — and suddenly it’s failing, producing wrong results, or worse, actioning on the same accounts again and again.</p><p>No obvious error. No clear reason. It worked perfectly five minutes ago.</p><p>If you’ve been in data long enough, you know that specific kind of panic. And if you haven’t experienced it yet, you will.</p><p>I’ve spent the last few years building and deploying data systems that run in production environments, touching millions of accounts and firing fraud strategies that are actioned on real customers in real time. What I’ve learned is that the gap between “it works on my machine” and “it works in production” isn’t really a technical gap. It’s a <em>thinking</em> gap.</p><p>This article is about closing that gap before you deploy — not after.</p><h3>The Uncomfortable Truth About Dev Environments</h3><p>Your dev environment is lying to you.</p><p>It’s stable, controlled, and nothing is competing for resources. You have clean data, the server is always up, and everything behaves exactly as expected. It’s a perfect little sandbox that makes you feel like a genius.</p><p>Production is none of those things.</p><p>Production is messy, shared, and completely indifferent to how elegant your code is. The first time you truly understand this is usually when something you built breaks in a way you never anticipated.</p><p>I learned this while running a machine learning model as part of a monthly campaign process. The model worked flawlessly in testing. But we were dependent on a Python server that was, to put it charitably, unreliable. Some days it was up. Some days it wasn’t. And when it went down mid-process, the entire monthly campaign stalled.</p><p>The business couldn’t wait. The campaign had to go out. And there I was, staring at a model that worked perfectly in every condition except the one that actually mattered.</p><p>That experience taught me the first rule of building for production: <strong>design for failure, not for success.</strong></p><h3>Rule 1: Design for Failure, Not for Success</h3><p>Most data work is built for exploration, not execution.</p><p>When you’re deep in analysis — trying to answer a question, validate a hypothesis, prove a point to a stakeholder — your entire focus is on getting the right answer. You’re not thinking about whether your code is efficient. You’re not thinking about how it will behave when it runs on a scheduler at 2 am against live data. You’re thinking about the output in front of you right now.</p><p>That’s fine. That’s what exploratory work is supposed to be.</p><p>The problem is when that exploratory code becomes the production code. When the notebook that answered the question becomes the process that runs every month. When nobody stopped to ask — <em>wait, should we rebuild this before we deploy it?</em></p><p>I’ve seen this happen more times than I can count. A process gets built to answer a one-time question, it works, someone says “great, let’s just schedule this,” and suddenly temporary code is touching millions of accounts every day. And when it breaks — and it will break — nobody fully understands it because it was never designed to be understood. It was designed to answer a question.</p><p>So when I faced the unstable server problem, fixing the server wasn’t an option — it was infrastructure outside my control. Instead, I removed the dependency entirely. I packaged the model using Python’s Pickle library, which serialized the entire model into a portable file that could run on any machine with minimal setup. No server required.</p><p>Suddenly, the monthly process was no longer tied to server availability. The campaign went out on time regardless of what the server was doing.</p><p>That’s designing for failure. Before you deploy anything, ask yourself: <em>what is the single point of failure here, and what happens when it breaks?</em> Then build the answer to that question into your design — before you deploy, not after.</p><h3>Rule 2: Scale Changes Everything</h3><p>There’s a version of your code that works at 10,000 rows. There’s a completely different version that needs to work at 10 million. And there’s an entirely different beast when you’re processing millions of records daily across systems that other teams are hitting simultaneously.</p><p>I ran into this with a process that loaded large datasets from one system into SQL Server. At small volumes it was fine. At scale it became a different story — jobs running for 1.5 hours, sometimes failing halfway through with transaction log errors. The team had lived with it so long it had become background noise. Just something you’d restart and wait on.</p><p>After digging into the configuration, I found a bulk insert parameter that had never been used in our environment. Implementing it correctly dropped that same process from 1.5 hours to 3 minutes.</p><p>Same logic. Same data. Same destination. The only thing that changed was understanding how the system actually behaved at scale.</p><p>Before you deploy, ask: <em>have I tested this at real volume?</em> Not a sample. The actual thing. Because scale doesn’t just slow processes down — it surfaces problems that simply don’t exist at small volumes.</p><h3>Rule 3: Complexity Is a Liability</h3><p>The more moving parts something has, the more ways it can fail — and the harder those failures are to find.</p><p>I once inherited a finance process that had accumulated years of complexity. What started as a straightforward calculation had grown into a 20+ step pipeline — legacy code branching everywhere, manual inputs at different stages, Excel files being loaded manually, dependencies nobody fully understood anymore.</p><p>It worked. Sometimes. When everything aligned perfectly, it produced the right output. But it was fragile. One manual step skipped, one file in the wrong format, one upstream change nobody communicated — and the whole thing would produce wrong numbers silently. No error. No alert. Just wrong output that someone might catch days later, or might not catch at all.</p><p>When I redesigned it, the goal wasn’t just speed. It was simplicity. Fewer steps. Automated inputs instead of manual ones. A single source of truth instead of five. Clear dependencies instead of implicit ones.</p><p>The redesigned version didn’t just run faster and use a fraction of the storage — it was fully automated with built-in logging and audit history. Every run was traceable. Every input was recorded. If something went wrong, you could go back and see exactly what happened, when it happened, and why. No more hunting through emails to figure out which Excel file someone loaded three weeks ago.</p><p>That’s the real win of reducing complexity. Complexity hides problems. Simplicity surfaces them.</p><p>Before you deploy, ask: <em>is every step in this process necessary?</em> If you can’t explain why a step exists, it will confuse the next person who touches it — and in production, confusion is expensive.</p><h3>Rule 4: Your Code Will Outlive Your Memory of Writing It</h3><p>Six months from now, someone else will need to modify what you built. That someone might be a colleague, a new team member, or, in the worst case , you, staring at your own code with no memory of why you made the decisions you did.</p><p>Write for that person.</p><p>Document the <em>why</em>, not just the <em>what</em>. Capture the business logic, not just the technical logic. Use variable names that mean something. Build a process that a reasonably competent analyst can follow without having to track you down.</p><p>In production environments, undocumented code doesn’t just create confusion — it creates risk. Someone modifies a parameter because it seemed arbitrary. A dependency gets removed because nobody knew it was load-bearing. A process that ran fine for two years suddenly starts producing wrong output because someone made a change they didn’t fully understand.</p><p>The best production code I’ve seen isn’t the cleverest. It’s the most readable. The code that a new team member can pick up in a day. The code that, when something breaks, someone can actually debug without reverse engineering your thought process from scratch.</p><h3>Before You Deploy: A Quick Checklist</h3><p><strong>Failure:</strong> What breaks this? What happens when it does? Have you built a fallback?</p><p><strong>Scale:</strong> Have you tested at real volume — including messy, malformed, late-arriving data?</p><p><strong>Complexity:</strong> Can you reduce the number of steps? Can you eliminate manual dependencies? Will failures be visible or silent?</p><p><strong>Readability:</strong> Can someone who didn’t write this understand it in 30 minutes? Is the business logic documented?</p><p><strong>Monitoring:</strong> How will you know when it breaks? (More on this in Part 3 — but start thinking about it now, because the best time to build monitoring is before you need it.)</p><h3>The Mindset Shift</h3><p>The difference between an analyst and someone who deploys reliable production systems isn’t really about technical skills. It’s about how you think about your work.</p><p>Analysts optimize for correctness. Production engineers optimize for correctness <em>and</em> resilience.</p><p>When you start asking “what happens when this breaks” before “does this work” — you’ve made the shift. When you design out single points of failure before someone asks you to, you’ve made the shift. When you write code for the person who comes after you instead of for the demo you’re about to give, you’ve made the shift.</p><p>Production will eventually test everything you build. The only question is whether you’ve thought about that before or after you deploy.</p><p><em>Part 2: Deploying Without Breaking Things — coming next week.</em></p><p><em>Part 3: Keeping it Alive: Monitoring Fraud Strategies in Production — coming in two weeks.</em></p><p><em>Have questions or want to share your own production stories? Connect with me on </em><a href="https://www.linkedin.com/in/mudita-garg/"><em>linkedin.com/in/mudita-garg</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44e43e1eca59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Popular Machine Learning Algorithms in Simple Terms, Part 2]]></title>
            <link>https://medium.com/@muditagarg12/popular-machine-learning-algorithms-in-simple-terms-part-2-867faf0f1906?source=rss-bddec522776a------2</link>
            <guid isPermaLink="false">https://medium.com/p/867faf0f1906</guid>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[clustering-algorithm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mudita Garg]]></dc:creator>
            <pubDate>Thu, 28 Sep 2023 00:49:11 GMT</pubDate>
            <atom:updated>2023-09-28T00:49:11.222Z</atom:updated>
            <content:encoded><![CDATA[<p>We discussed supervised learning algorithms in <a href="https://medium.com/@muditagarg12/popular-machine-learning-algorithms-in-simple-terms-part-1-634c96d1d74a">Part 1</a>. Here in part 2, we will learn unsupervised machine learning algorithms in simple terms.</p><h3>K-Means Clustering</h3><p>Imagine you have a big room full of scattered toys, and you’re blindfolded. You want to group these toys into piles, but since you can’t see them, you decide on a number (let’s say 3 piles) and then try to gather toys that feel similar into each bank. After a few tries, you might end up with piles of soft, plastic, and metal toys. This is how K-Means works but with data points instead of toys. K-means is used for grouping similar data points into K number of groups. The goal is to find patterns or similarities in your data, even when you don’t have predefined categories or labels.</p><p><strong>Key Concepts</strong></p><ol><li>Clusters: These are the groups or piles in which data is categorized.</li><li>Centroids: Think of these as the center or representative point of each cluster. If it were the toy example, it’s like picking one toy from each pile that feels the most “average” for that pile.</li><li>K: This is the number of clusters you decide to have (like choosing to make 3 piles). You have to decide on ‘K’ beforehand.</li></ol><p><strong>How does it work?</strong></p><ol><li><strong>Initialization:</strong> You can think of it as sorting marbles. First, you randomly pick a few marbles from the bag (let’s say 3) and put them aside. These marbles represent the initial groups, and we call them cluster centers.</li><li><strong>Assignment:</strong> You can think of it as matching colors. You go through all the marbles in the bag and check which of the 3 cluster centers is closest to each marble based on their colors. Each marble goes to the closest cluster center.</li><li><strong>New Centers/Update: </strong>Now, you look at the marbles in each group (cluster) and find the one in the middle (by finding the average of all data points assigned to each cluster). This middle marble becomes the new cluster center for that group.</li><li><strong>Repeating Steps:</strong> You keep matching and finding new center steps over and over until no marbles want to change groups. This means you have found the best groupings you can.</li><li><strong>Result:</strong> At the end, you would have grouped all your marbles by color, even though you didn’t know how many colors there were initially.</li></ol><p><strong>Type of Data<br></strong>K-Means works well with numeric data where the concept of similarity is based on Euclidean Distance.</p><p><strong>Real-life Example<br></strong>Imagine you have a dataset of customer purchase histories (e.g., what they buy and how often). You want to group customers based on their shopping habits, but you don’t know in advance how many groups there are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/0*F_Kdiz-f1ZsBvRVT" /><figcaption>Fig 1: Example of K-Means Clustering</figcaption></figure><p><strong>When to use K-Means?</strong></p><ol><li>When you want to discover and explore natural groupings or patterns within your data.</li><li>Customer Segmentation: Businesses can use K-Means to group customers with similar purchase behaviors, helping tailor marketing strategies and product recommendations.</li><li>Image Compression: In image processing, K-Means can reduce the storage space needed for images. By clustering similar pixel colors and replacing them with cluster centers, you can achieve compression while maintaining visual quality.</li><li>Anomaly Detection: K-Means can identify outliers or anomalies in datasets. In cybersecurity, it’s used to detect unusual network behavior or fraudulent transactions by clustering normal and abnormal patterns.</li></ol><p><strong>Advantages of K-Means Clustering</strong></p><ol><li>Simplicity: It is easy to understand and implement, making it accessible.</li><li>Speed and Scalability: It is computationally efficient and can handle large datasets and many features. This makes it suitable for real-time or big-data applications.</li><li>Quantifiable Metrics: You can use metrics like Within-Cluster Sum of Squares (WCSS) or Silhouette score to evaluate the quality of clustering objectively.</li><li>Widely Supported: It is available in many machine learning libraries and software packages, making it easily accessible to researchers and practitioners.</li></ol><p><strong>When not to use K-Means?</strong></p><ol><li>Non-Spherical or Irregularly Shaped Clusters: If your groups don’t look like circles and are different sizes, K-Means may not work well.</li><li>Sensitive to Initialization: It is sensitive to the initial placement of cluster centers. Different initializations can lead to different results. It is like sorting cards differently each time you play a game.</li><li>Outlier Sensitivity: K-Means treat outliers as a part of the nearest cluster, which may not be suitable for tasks where you want to identify and separate outliers from the main groups.</li><li>Categorical or Binary Data: It primarily works with numerical data, relying on distance metrics.</li></ol><h3>Hierarchical<strong> Clustering</strong></h3><p>It is another clustering algorithm for grouping data points into nested clusters, creating a tree-like structure called a dendrogram. Imagine you’re trying to arrange a set of books. Instead of sorting them by just one theme, you start by pairing books that are super similar. Maybe two detective novels first. Then, you group similar pairs into larger groups, say detective novels with thriller novels. As you keep going, you’ll end up with a big group, like fiction vs. non-fiction. Hierarchical Clustering works similarly, creating a tree of clusters. It organizes data hierarchically, allowing you to explore clusters at different levels of granularity.</p><p><strong>Key Concepts</strong></p><ol><li>Dendrogram: A tree-like diagram that shows the arrangement of the clusters. It’s like the family tree for your data.</li><li>Agglomerative &amp; Divisive: Two main approaches. Agglomerative starts with each data point as its own cluster and pairs them up step by step (bottom-up). Divisiveness starts with one big cluster and breaks it down (top-down).</li><li>Linkages: The method used to determine the ‘distance’ between clusters. For example, using the closest points, farthest points, or an average.</li></ol><p><strong>How does it work?</strong></p><ol><li><strong>Start with Individuals:</strong> Initially, each animal is its own group. Think of them as individual leaves on a tree.</li><li><strong>Find similar Pairs:</strong> You look at all pairs of animals, decide which two are the most similar to each other, and merge them into larger groups. You keep doing this until everything is in one big group at the top of the tree.</li><li><strong>Dendrogram:</strong> This results in a tree called a dendrogram. The result is like a family tree, showing how animals are related to each other. You can look at broad categories (like all mammals) or go deeper (like grouping specific types of reptiles together) by cutting trees at different levels.</li></ol><p><strong>Type of Data</strong></p><ol><li>Qualitative Data: Hierarchical clustering is versatile and can handle non-numeric data, such as text or categorical data.</li><li>Nested Data: Data that naturally have hierarchical relationships, like product categories or biological taxonomies.</li></ol><p><strong>Real-life Example<br></strong>Suppose you’re a librarian, and you want to categorize books not just by genre but also sub-genres, themes, and so on. You could use hierarchical clustering to create a multi-level classification system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*ZqnM1HhqTiZat677.png" /><figcaption>Fig 2: Example of Hierarchal Clustering</figcaption></figure><p><strong>When to use Hierarchical Clustering?</strong></p><ol><li>Taxonomy Creation: Classifying animals or plants into a hierarchical taxonomy.</li><li>User Profiling: Understanding user behavior in multiple dimensions.</li><li>Content Structuring: Organizing articles or products in a multi-level category structure.</li></ol><p><strong>Advantages of Hierarchical Clustering</strong></p><ol><li>No Need for ‘K’: You don’t need to specify the number of clusters in advance.</li><li>Rich Output: Provides a dendrogram, which offers more than just clusters, such as the relationships and hierarchy between them.</li><li>Interpretable: The dendrogram is easy to understand, offering a visual representation of the clustering process.</li></ol><p><strong>When Not to Use Hierarchical Clustering?</strong></p><ol><li>Large Datasets: Not suitable for large datasets due to computational expense and less efficiency.</li><li>Ambiguity: The dendrogram might be hard to cut when clusters aren’t distinct.</li><li>No Backtracking: Once a decision is made to merge two clusters, it can’t be undone.</li><li>Flat Clustering: If you’re only interested in a single level of clustering, other methods might be more efficient.</li></ol><h3>DBSCAN</h3><p>DBSCAN is like having a big party where people naturally gather in groups based on common interests. Instead of deciding how many groups you’ll have in advance, you let the crowd form groups naturally. You identify dense areas where people have gathered and mark them as groups. Those who don’t fit into any group are considered outliers.</p><p><strong>Key Concepts:</strong></p><ol><li>Density: A cluster is a dense region of data points separated by low-density regions from other dense regions.</li><li>Epsilon (ε): This is the radius of the neighborhood around each data point. It specifies how close points should be to form a cluster.</li><li>MinPts: The minimum number of points required to form a dense area.</li></ol><p><strong>How Does it Work?</strong></p><ol><li>Choose a Point: Randomly pick a point in the dataset.</li><li>Find Neighbors: Identify all points within a radius of ε from the chosen point.</li><li>Form Cluster or Noise: If there are at least MinPts within the radius, start forming a cluster around the point. If not, mark it as noise (which can later become part of a cluster).</li><li>Expand Cluster: Add nearby points of the initial cluster point to the cluster and search their ε-neighborhood. Repeat this process until no new points can be added.</li><li>Iterate: Go back to step 1 until all points have been visited.</li></ol><p><strong>Type of Data</strong></p><p>DBSCAN works well with arbitrary shapes of clusters and is good for data with noise and outliers.</p><p><strong>Real-life Example</strong></p><p>Imagine a shopping mall where people tend to gather around their interests: fashion, tech gadgets, or food. DBSCAN would identify these gatherings as clusters and would also point out people who are just wandering around as outliers.</p><p><strong>When to Use DBSCAN?</strong></p><ol><li>Anomaly Detection: Spotting fraud, defects, or outliers in general.</li><li>Cluster Shapes: When clusters are irregularly shaped, unlike circles or spheres.</li><li>No Cluster Count: When you don’t know how many clusters to expect.</li></ol><p><strong>Advantages of DBSCAN</strong></p><ol><li>No Need for ‘K’: Unlike K-Means, you don’t need to specify the number of clusters.</li><li>Handles Noise: Can detect outliers, making it good for anomaly detection.</li><li>Arbitrary Shapes: Can find arbitrarily shaped clusters.</li></ol><p><strong>When Not to Use DBSCAN?</strong></p><ol><li>Variable Density: Struggles with clusters of varying densities.</li><li>High-Dimensionality: This may not perform well when the number of dimensions is very high.</li></ol><h3>Principal Component Analysis (PCA)</h3><p>Principal Component Analysis (PCA) is like taking a bulky set of keys and replacing it with a simplified, lightweight key fob. Imagine you have data in three dimensions (like a room with width, length, and height). PCA helps you understand which dimensions (features) carry the most information, allowing you to simplify or “flatten” your data into fewer dimensions while retaining its essence.</p><p><strong>Key Concepts:</strong></p><ol><li>Components: These are the new dimensions that are linear combinations of the original features. They’re ranked by the amount of variance they capture from the original data.</li><li>Dimensionality Reduction: The process of reducing the number of random variables and keeping only those that are most important.</li></ol><p><strong>How Does it Work?</strong></p><ol><li>Standardize Data: First, the data is standardized to have a mean of zero and a standard deviation of one.</li><li>Covariance Matrix: Compute the covariance matrix to understand how different features relate to each other.</li><li>Eigenvalues &amp; Eigenvectors: These are calculated from the covariance matrix, and they form the new dimensions.</li><li>Sort and Select: Eigenvectors are sorted by their eigenvalues, and only the top eigenvectors are selected to form the new dimensions.</li></ol><p><strong>Type of Data<br></strong>PCA is generally used on numerical data where the aim is to understand the underlying structure or to reduce the number of variables while preserving as much information as possible.</p><p><strong>Real-life Example<br></strong>Let’s say you work in a wine shop and have information on several characteristics of different wines — like acidity, sugar content, and alcohol percentage. It’s hard to compare all these factors at once. You can use PCA to distill these characteristics into two or three main components that capture the essence of the differences between the wines. Then, you could more easily recommend wines to customers based on these simplified categories.</p><p><strong>When to Use PCA?</strong></p><ol><li>When you have a large number of correlated features.</li><li>Before applying machine learning algorithms to simplify the model and speed up the training process.</li></ol><p><strong>Advantages of PCA</strong></p><ol><li>Dimensionality Reduction: PCA helps in reducing the number of features in a dataset without losing much information. This simplification is often critical for both speeding up other machine learning algorithms and making them perform better.</li><li>Less Noise: By focusing only on the principal components that account for the most variance in the data, PCA helps in eliminating noise. This can improve the performance of subsequent models applied to the transformed dataset.</li><li>Improved Visualization: Reducing the number of dimensions with PCA can make it much easier to visualize complex datasets, helping to identify patterns or clusters in the data that might not have been apparent in the high-dimensional space.</li></ol><p><strong>When Not to Use PCA?</strong></p><ol><li>The interpretability of features is crucial, as PCA’s components are combinations of the original features and may be hard to interpret.</li></ol><h3>Autoencoders</h3><p>Autoencoders are like self-taught artist who learns to simplify complicated scenes into easier-to-understand sketches. They can learn to encode and decode complex data, making them good for dimensionality reduction or anomaly detection.</p><p><strong>Key Concepts:</strong></p><ol><li>Encoder: The part of the network that compresses the input into a latent-space representation.</li><li>Decoder: The part of the network that reconstructs the input data from the internal representation.</li><li>Latent Space: The simplified, compressed representation of the input data.</li></ol><p><strong>How Does it Work?</strong></p><ol><li>Train the Encoder: Feed data into the encoder, which compresses it into a latent space.</li><li>Train the Decoder: Use the latent space data to reconstruct the original data as closely as possible.</li><li>Optimize: Tweak the encoder and decoder to minimize the difference between the input and the reconstructed output.</li></ol><p><strong>Type of Data<br></strong>Autoencoders work well with complex data types like images, audio, and text. They can also be applied to numerical data for tasks like anomaly detection.</p><p><strong>Real-life Example<br></strong>Imagine you work for a hospital, and you have MRI scans of your brain. These scans are large and detailed, so they take up a lot of storage space and are hard to analyze quickly. Autoencoders can “compress” these images into a simpler form that still captures the key features necessary for diagnosis. Later, these compressed images can be “decompressed” to closely approximate the original scans for further medical analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pJeI8V2v4prHsJ9Y.png" /></figure><p>PCA and Autoencoders help simplify complex data but are most beneficial when used on the types of data they are best suited for.</p><p><strong>When to Use Autoencoders?</strong></p><ol><li>For feature learning and dimensionality reduction.</li><li>For anomaly detection, the autoencoder is trained on normal data, and anomalies are detected based on reconstruction error.</li></ol><p><strong>Advantages of Autoencoders</strong></p><ol><li>Feature Learning: Autoencoders are particularly good at automatically learning representations from the data. This is useful when you’re trying to discover an informative representation of the input data without using any labels.</li><li>Noise Reduction: Autoencoders can be configured to remove noise from data. This is valuable in scenarios like image denoising where you want to extract useful information from an image while ignoring irrelevant details caused by factors such as camera artifacts.</li><li>Data Compression: By learning an efficient encoding, autoencoders can serve as a data compression algorithm. The encoding captures the most important features of the data, allowing you to store or transmit it in a more compact form without losing too much information.</li></ol><p><strong>When Not to Use Autoencoders?</strong></p><ol><li>A simpler technique like PCA can achieve similar results with less computational effort.</li><li>When interpretability is a priority. Autoencoders can be more challenging to understand compared to simpler methods.</li></ol><h3>Association Rule Learning</h3><p>In essence, Association Rule Learning is like a clever detective in a shopping scene. It examines customer transactions to uncover patterns, such as frequently bought items that appear together in the same cart. The algorithm sifts through vast amounts of data to detect these recurring trends.</p><p><strong>Key Concepts:</strong></p><ul><li>Support: This is the frequency with which items appear together in the dataset. For example, if bread and butter frequently appear together in shopping carts, they have high support.</li><li>Confidence: This measures how often items in a rule appear together. For instance, if bread is purchased, what is the likelihood that butter will also be purchased? High confidence means it’s highly likely.</li><li>Lift: Lift indicates the strength of a rule over random occurrence. A lift greater than 1 usually implies that the rule is useful for predicting the occurrence of the outcome.</li></ul><p><strong>How Does it Work?<br></strong>The algorithm scans customer transaction data to find frequent item sets. Then, based on these item sets, it generates rules that describe how often items are bought together. Think of it like a puzzle-solver looking at every customer cart, jotting down common pairs or sets of items, and then making smart guesses about future purchases.</p><p><strong>Type of Data<br></strong>Association Rule Learning is typically applied to transactional data — basically lists of items bought by customers. This could be anything from online retail purchases to market basket data in grocery stores.</p><p><strong>Real-life Example<br></strong>Suppose you own an online pet supply store. By analyzing past transactions, you discover that people who buy dog food often also buy dog toys. As a strategic move, you decided to offer a bundle package containing both dog food and a dog toy at a discounted price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*Wa0dQtNPjoTINaPtBuW3nQ.png" /></figure><p><strong>When to Use Association Rule Learning?<br></strong>Use it when you’re trying to find hidden patterns or associations in a large dataset. For instance, retailers often use it to determine product placement and promotion strategies.</p><p><strong>Advantages of Association Learning Algorithm</strong></p><ol><li>Actionable Insights: These rules often result in straightforward, actionable insights for businesses.</li><li>Easy to Understand: Even non-experts can understand the rules and the logic behind them, making it easier to implement changes based on these rules.</li><li>Versatile: Association Rule Learning can be applied in various fields, not just retail. It can be used in healthcare, for example, to find associations between different types of medication and adverse effects.</li></ol><p><strong>When Not to Use Association Rule Learning?<br></strong>Avoid using it when the goal is to make predictive models that require labeled outcomes. Association Rule Learning is about uncovering interesting relations in the data, not predicting a specific outcome based on input features.</p><p>Thank you for reading this series on machine learning methods. Please share this with your friends and colleagues and follow for more content!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=867faf0f1906" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Popular Machine Learning Algorithms in Simple Terms, Part 1]]></title>
            <link>https://medium.com/@muditagarg12/popular-machine-learning-algorithms-in-simple-terms-part-1-634c96d1d74a?source=rss-bddec522776a------2</link>
            <guid isPermaLink="false">https://medium.com/p/634c96d1d74a</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mudita Garg]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 04:07:46 GMT</pubDate>
            <atom:updated>2023-09-12T04:07:46.887Z</atom:updated>
            <content:encoded><![CDATA[<p>In our increasingly digital world, the term “Machine Learning” seems to pop up everywhere. From recommending movies on Netflix to self-driving cars, machine-learning algorithms are behind many technological wonders of our time. But how exactly do these work? In this article, we will explore some of the most commonly used Machine Learning algorithms and provide real-world examples to help you understand them better.</p><h3><strong>Linear Regression</strong></h3><p>Linear Regression is a type of supervised learning algorithm used in statistics and machine learning. It aims to model and analyze the relationships between a dependent variable and one or more independent variables. The main idea is to fit a linear equation to observed data. The simplest form of the linear regression equation with two variables is represented as :</p><p><strong><em>Y</em>=<em>aX</em>+<em>b</em></strong></p><p>Here:</p><ul><li><em>Y</em> is the dependent variable (what we’re trying to predict).</li><li><em>X</em> is the independent variable (the input).</li><li><em>a</em> and <em>b</em> are coefficients that are determined through the learning process.</li></ul><p><strong>Type of Data</strong><br>Linear Regression requires a continuous dependent variable and can work with continuous and categorical independent variables. The relationship between the dependent and independent variables should be linear.</p><p><strong>Real-life Example</strong><br>Imagine you want to predict a student’s final grade based on their hours of study. If you plot each student’s study hours (independent variable) against their grades (dependent variable), linear regression would try to draw a straight line that best fits the data points. Using this line, you can then predict the grade of a student based on the number of hours they’ve studied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*zcTKJHfcNK49BgrUPtdSVQ.png" /><figcaption>Fig 1: Linear Regression for predicting grades of a student</figcaption></figure><p><strong>When to use Linear Regression?</strong></p><ol><li>Predicting sales amounts based on advertising spend.</li><li>Estimating house prices based on square footage and number of bedrooms.</li><li>Forecasting crop yields based on rainfall and temperature.</li></ol><p><strong>When Not to Use Linear Regression?</strong></p><ul><li>Non-linear Relationships: Not all relationships are linear. If the relationship between variables is more complex (e.g., quadratic or exponential), linear regression won’t capture it well.</li><li>Categorical Outcomes: If the outcome or dependent variable is categorical (e.g., “Yes” or “No”, “Cat” or “Dog”), logistic regression or other classification algorithms might be more appropriate.</li><li>Highly Skewed Distributions: If the residuals (the differences between observed and predicted values) don’t follow a normal distribution or if there’s a high variance among them, the assumptions of linear regression are violated.</li></ul><h3>Logistic Regression</h3><p>Imagine you’re trying to predict if it’ll rain tomorrow. You can’t just say it’ll rain 5 or -3 (negative rain?). Instead, you’d prefer to say there’s a 70% chance of rain. This prediction is between 0% (no rain) and 100% (definite rain). Logistic Regression helps in such situations where the outcome is between two possibilities.</p><ol><li><strong>Sigmoid Curve: </strong>At the heart of Logistic Regression is this special S-shaped curve. It takes any input and produces a value between 0 and 1 — perfect for probabilities!</li><li><strong>Odds Ratio:</strong> It’s the odds of an event happening in comparison to it not happening. Like saying the odds of it raining tomorrow are 2:1.</li></ol><p><strong>How Does It Work?</strong></p><ol><li>Input Data: Similar to other regression techniques, you have input variables. Let’s stick with the rain example. Inputs could be humidity, cloud cover, wind speed, etc.</li><li>Calculate Probabilities: Using these inputs, Logistic Regression computes a value. Instead of a straight line (as in Linear Regression), it uses the Sigmoid Curve to ensure the value is between 0 and 1.</li><li>Classify: Based on the probability, you classify the outcome. For instance, if you have a threshold of 50%, anything above is “Will Rain” and below is “Won’t Rain.”</li></ol><p><strong>Type of Data</strong> <br>We use Logistic Regression when our outcome (what we want to predict) is categorical and specifically binary. “Binary” means it has just two possible outcomes, like “Yes” or “No”, “True” or “False”, “Pass” or “Fail”.</p><p><strong>Real-life Example</strong><br>Let’s use the previous example of predicting a student’s final grade (Scores), but now we only have their scores and whether the student passed or not (GPA Pass). This is now a probability situation. See the below graph, which tells the student has a 0 to 1 probability of passing based on their grades.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_asf1kfMkhfxzBpqBbHOZQ.png" /><figcaption>Fig 2: Logistic Regression for predicting if a student will pass or not</figcaption></figure><p><strong>When to Use Logistic Regression?</strong></p><ol><li>Predicting if a patient has a disease (like COVID-19) based on their symptoms.</li><li>Guessing if a student will pass or fail based on their attendance.</li><li>Estimating if a bank customer will default on a loan or not based on their credit history.</li><li>Probability Matters: When you’re more interested in the probability of occurrence of an event rather than its exact prediction.</li></ol><p><strong>When Not to Use Logistic Regression?</strong></p><ul><li>Multiple Categories: If you’re trying to predict something with more than two outcomes (like if a customer’s favorite fruit is an apple, banana, or cherry), Logistic Regression isn’t the best tool. For that, we might use other techniques like “Multinomial Logistic Regression” or different algorithms.</li><li>Continuous Outcomes: If you’re trying to predict a number (like the price of a house or a car’s speed), that’s a job for Linear Regression, not Logistic.</li></ul><h3>Decision Trees</h3><p>At its core, a Decision Tree is a flowchart-like tree structure. It breaks down a dataset into smaller and smaller subsets, making a decision at every level. Each node in the tree represents a feature (or attribute), each branch represents a decision rule, and each leaf represents an outcome.</p><p><strong>Entropy:</strong> It’s a measure of randomness or unpredictability. High entropy means more mixed-up data and low entropy means more organized data.</p><ul><li>Imagine you have a jar filled with red and blue marbles. If the jar has all red or all blue marbles, then there’s no confusion or “disorder” in picking one; you know what you’ll get. This is <strong><em>low entropy</em></strong>. Now, if the jar is a mix of red and blue marbles, there’s a lot of uncertainty about what you’ll pick. This situation has <strong><em>high entropy</em></strong>.</li><li>In decision trees, entropy measures the randomness or uncertainty in the data. The goal is to organize or sort the data in such a way that the uncertainty (entropy) decreases.</li></ul><p><strong>Information Gain:</strong> The reduction in entropy is achieved by partitioning a dataset based on an attribute. A decision tree will try to partition data on the attribute that results in the largest information gain.</p><ul><li>Let’s stay with the jar analogy. Suppose you have a magic sieve that can separate the marbles based on some property, maybe size. If after using the sieve, one jar has mostly red marbles and another has mostly blue, then the sieve did a good job in reducing the randomness.</li><li>Information gain measures how effective a feature is in separating the data. It’s the difference in entropy before and after the data is divided using a particular feature. The feature that provides the highest information gain will be the one used to split the data.</li><li>In simple terms, information gain tells us how much “information” a feature gives us about the class. It’s the reduction in uncertainty achieved by partitioning a dataset based on that feature.</li></ul><p><strong>How Does it Work?</strong></p><ol><li>Selection of attribute: The attribute to be placed at the root or internal node is selected using statistical methods.</li><li>Branching: Based on the attribute’s value, the dataset is split into subsets. This process is repeated recursively, resulting in a tree-like model of decisions.</li><li>Stopping Criteria: The division continues until one of the stopping conditions is met, such as the subset at a node all belonging to the same category, or reaching a maximum tree depth.</li></ol><p><strong>Type of Data </strong><br>Decision Trees can be used for both numerical (like predicting prices) and categorical outcomes (like Yes or No answers).</p><p><strong>Real-life Example</strong><br>Consider an example where you want to predict if a person will play tennis based on the weather. The Decision Tree might start with a question like “Is it sunny?” and then based on the answer Yes or No, it then branches off into “Is the humidity high?” Based on the sequence of answers, the tree will conclude whether the person will play tennis or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vxgwRj2bNeAdmTADt97d0Q.png" /><figcaption>Fig 3: Decision Tree on playing tennis based on weather</figcaption></figure><p><strong>When to Use Decision Trees?</strong></p><ol><li>When deciding if a person is a good fit for a job based on their skills and experience.</li><li>Predicting if it will rain tomorrow based on humidity, cloud cover, and other factors.</li><li>Classifying animals based on features like number of legs, size, and habitat.</li></ol><p><strong>Advantages of Decision Trees</strong></p><ol><li>Interpretable &amp; Transparent: They are easy to understand and visualize, even for non-experts.</li><li>Minimal Data Prep: They require relatively little effort for data preparation, and they can handle both numerical and categorical variables.</li><li>Non-parametric: They make no assumptions about the functional form of the data.</li></ol><p><strong>When Not to Use Decision Trees?</strong></p><ul><li>Overly Complex Trees: If there are too many questions and the tree becomes too big, it might make very specific decisions that don’t generalize well to new situations.</li><li>Continuous Data: While Decision Trees can handle numerical data, they might not be the best choice for very intricate continuous data predictions.</li></ul><h3>Random Forest</h3><p>A Random Forest is a collection (or ensemble) of individual decision trees that operate as a committee. Random Forest builds multiple decision trees and then merges their decision. If one Decision Tree is like one person’s opinion, then a Random Forest is like asking a whole group of people for their opinions and going with the majority vote. Here, each expert is like a decision tree, and the collective decision represents the forest.</p><ul><li><strong>Out-of-Bag (OOB) Error:</strong> Since only around two-thirds of the data is used to train each tree (due to the bootstrap process), the remaining one-third, not used in the training, can be used to validate. This helps in estimating the accuracy of the forest.</li><li><strong>Feature Importance:</strong> Random Forest can rank the importance of input features. Features used frequently at the top of the trees across many trees are often more significant for prediction.</li></ul><p><strong>How Does it Work?</strong></p><ol><li>Bootstrap Aggregating (Bagging): The algorithm creates multiple subsets of the original dataset by randomly selecting samples with replacements. This means some samples may appear more than once in a subset, while others might not appear at all.</li><li>Building Trees: For each of these subsets, a decision tree is grown. Unlike a standard decision tree, where the best split among all variables is chosen for each node, each tree in a Random Forest picks the best split among a random subset of the features. This randomness ensures that the individual trees are uncorrelated with each other.</li><li>Aggregation: For a classification problem, each tree in the forest predicts a class, and the class with the most votes wins. For a regression problem, the average prediction of all the trees is the final output.</li></ol><p><strong>Type of Data </strong><br>Random Forest can handle both numerical and categorical data.</p><p><strong>Real-life Example</strong><br>Consider a medical diagnosis system trying to detect if a patient has a particular disease. Various tests and examinations have been conducted, resulting in data points like heart rate, blood pressure, age, BMI, etc. Rather than relying on the judgment of one test or one doctor, imagine consulting a team of doctors (trees) that each specializes in specific tests. Some doctors might only look at heart rate and age, while others might focus on BMI and blood pressure. After analyzing, if the majority of the doctors conclude the patient has the disease, the final diagnosis is positive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*EKzWWboWE4NlExz1.jpg" /><figcaption>Fig 4: Random Forest framework</figcaption></figure><p><strong>When to Use Random Forest?</strong></p><ol><li>Predicting if a product will be popular based on features like price, advertisement spend, and competitor products.</li><li>Classifying emails as spam or not spam.</li><li>Estimating disease outbreaks based on various health indicators.</li></ol><p><strong>Advantages of Random Forest</strong></p><ol><li>Accuracy: Often delivers very accurate predictions.</li><li>Handling Missing Data: Can handle missing values by either using median values to replace continuous variables or computing the proximity-weighted average of missing values.</li><li>Flexibility: This can be used for both regression and classification tasks.</li></ol><p><strong>When Not to Use Random Forest?</strong></p><ul><li>Complexity: A large number of trees can make the model slow and ineffective for real-time predictions.</li><li>Overfitting: For datasets with noise (unrelated random variations), it can overfit.</li><li>Black Box: While it’s easy to get predictions from a Random Forest, it can be challenging to interpret the inner workings. However, this is more interpretable than many other algorithms, like deep neural networks.</li></ul><h3>Support Vector Machines (SVM)</h3><p>SVM finds the best boundary (or hyperplane) that separates different classes in a dataset. Picture two types of animals in a field, say sheep and goats, intermingling. Your job is to put up a fence to separate them. SVM doesn’t just put up any fence; it ensures the fence is positioned where the gap between the two groups is widest. This means it finds the best boundary that separates different classes in a dataset.</p><ul><li>Linear &amp; Non-Linear SVM: In a 2D space, a linear SVM would separate data using a straight line, while a non-linear SVM might use a curved boundary.</li><li>Kernel Methods: These are mathematical functions (like the Polynomial kernel, Radial Basis Function, etc.) that help SVM in finding a hyperplane in the transformed space.</li></ul><p><strong>How Does it Work?</strong></p><ol><li>Finding the Hyperplane: Imagine all your data points as stars in the night sky. SVM tries to draw the widest possible “galactic highway” between these stars while keeping them apart based on their class. This “highway” is your hyperplane.</li><li>Support Vectors: The data points (or stars) that lie closest to the hyperplane and influence its orientation and location are called support vectors. They’re the critical elements of your dataset.</li><li>Margin Maximization: The goal of SVM is to maximize the margin, which is the distance between the hyperplane and these support vectors.</li><li>Kernel Trick: For data that isn’t linearly separable (i.e., you can’t draw a straight line to separate them), SVM uses a trick. It transforms the data into a higher dimension where it becomes linearly separable, thanks to functions called “kernels.”</li></ol><p><strong>Kernels:</strong> Imagine a city skyline where residential and commercial buildings are intermixed, and you’re trying to separate them using a simple boundary on a flat map. It seems impossible because they’re intertwined. However, consider the height of these buildings as an added dimension: commercial buildings might tower over residential ones. By taking into account this height (a third dimension), you can more easily distinguish the skyscraper-filled commercial zones from the shorter residential areas. In machine learning, the kernel is like introducing that height aspect; it transforms the data by adding dimensions, making intertwined points distinct and easier to separate.</p><p><strong>Real-life Example</strong><br>Imagine a fruit vendor who receives a mix of apples and oranges, but they’re all mixed up. Using an SVM, we can use features like fruit color and size to distinguish between apples and oranges. If apples are generally red and of a certain size and oranges are generally orange and larger, the SVM will try to find the best dividing line or boundary where on one side you’d mostly find apples and on the other side oranges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GfYBaFSnQBG5Y0yj.png" /><figcaption>Fig 5: SVM algorithm on classifying apples and oranges</figcaption></figure><p><strong>When to use SVM?</strong></p><ol><li>Face detection in images: Identifying if an image contains a face or not.</li><li>Text categorization: Labeling documents as one category or another.</li><li>Handwriting recognition: Classifying different handwritten characters or digits.</li></ol><p><strong>Advantages of SVM</strong></p><ol><li>Effective in High Dimensions: SVM works well when there are a lot of features or attributes.</li><li>Memory Efficient: Only uses a subset of the training points (support vectors) in the decision function.</li><li>Versatility: The choice of kernel function can make it versatile for various problems.</li></ol><p><strong>When not to use SVM?</strong></p><ul><li>Sensitive to Noise: A small number of mislabeled examples can drastically affect the hyperplane.</li><li>Not Natively Multi-Class: While it’s perfect for binary classification, multi-class problems require multiple SVMs.</li><li>Complexity and Speed: Training can be computationally intensive, especially for large datasets.</li></ul><p>In our next part, we will dive into some popular unsupervised learning algorithms in simple terms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=634c96d1d74a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>