<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhishekar Reddy on Medium]]></title>
        <description><![CDATA[Stories by Abhishekar Reddy on Medium]]></description>
        <link>https://medium.com/@abhishekareddy1234?source=rss-f2e9f4dbcfd0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*iP5kIi0kO63XTnSc</url>
            <title>Stories by Abhishekar Reddy on Medium</title>
            <link>https://medium.com/@abhishekareddy1234?source=rss-f2e9f4dbcfd0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:36:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abhishekareddy1234/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Comparative Study of High-Cardinality Encoding Techniques: Which One Works Best?]]></title>
            <link>https://medium.com/@abhishekareddy1234/a-comparative-study-of-high-cardinality-encoding-techniques-which-one-works-best-2bc2c2405404?source=rss-f2e9f4dbcfd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bc2c2405404</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[feature-encoding]]></category>
            <category><![CDATA[categorical-data]]></category>
            <dc:creator><![CDATA[Abhishekar Reddy]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 09:28:35 GMT</pubDate>
            <atom:updated>2026-11-19T09:28:35.370Z</atom:updated>
            <content:encoded><![CDATA[<p>I tested eight ways to turn thousands of categories into numbers on a ride-hailing dataset. The winner surprised me, and the accuracy numbers nearly fooled me.</p><h3>1. Introduction</h3><h3>What is categorical data?</h3><p>Categorical data is data that falls into labels instead of numbers: a driver’s ID, a city, a vehicle type, a pin code. You can’t add two pin codes or average two drivers, so these values don’t behave like numbers.</p><h3>What is categorical encoding, and why do we need it?</h3><p>Machine learning models do arithmetic. They multiply, split and compare numbers. So before a model can use a column like driver_id, we have to convert the labels into numbers. That conversion is called <strong>categorical encoding</strong>.</p><h3>What is high cardinality?</h3><p><strong>Cardinality</strong> is the number of unique values in a column. A column such as vehicle_model with 5 values is low-cardinality. A column such as pickup_zip with 1,500 values is <strong>high-cardinality</strong>. There is no hard cutoff, but once the unique values run into the hundreds or thousands, the usual tricks start to break down.</p><h3>Why is it a challenge?</h3><p>The classic encoder, One-Hot Encoding, creates one new column per category. With 600 drivers and 1,500 + 1,500 zip codes, that is 3,600 new columns, most of them almost entirely zeros. This makes training slower and the model more prone to memorising noise. And the other encoders each trade something away: some lose information, and some leak the answer into training.</p><p>So I ran an experiment to see what actually works.</p><h3>2. Understanding the Dataset</h3><p>The dataset describes ride-hailing operations: rides, drivers, vehicles and customer complaints. It has <strong>30,000 rows</strong> and no missing values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NbEQWQXIuUm6WFDD6zGe9Q.png" /></figure><p><strong>Target variable:</strong> complained is 1 if the customer raised a complaint and 0 otherwise. Only <strong>about 16%</strong> of rides have a complaint, so this is an imbalanced binary classification problem.</p><p><strong>High-cardinality columns:</strong> driver_id, pickup_zip and dropoff_zip. These need special encoding because One-Hot Encoding would add thousands of sparse columns. vehicle_model has only 5 values, so I One-Hot encoded it in every experiment to keep the comparison fair.</p><p>That 16% matters a lot later: a model that always says “no complaint” would already be about 84% accurate.</p><h3>3. Encoding Techniques Explained</h3><p>I compared eight approaches. For each one: what it is, how it works, a small example, and where it fits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SfPmREV_Lx-IXFIT4NfgBA.png" /></figure><h3>3.1 Capped One-Hot Encoding</h3><p><strong>Definition:</strong> One-Hot Encoding with a limit on how many categories get their own column.</p><p><strong>How it works:</strong> Only the most frequent categories (up to max_categories) get a column. All the rest are merged into a single &quot;infrequent&quot; bucket. With handle_unknown=&quot;infrequent_if_exist&quot;, categories never seen in training are sent to that same bucket instead of causing an error.</p><p><strong>Example:</strong> If a column has 1,500 zips and max_categories=20, the 19 most common zips get a column each. The other 1,481 share one column.</p><p><strong>Advantages:</strong> Simple, no target information used, so no leakage.</p><p><strong>Limitations:</strong> Throws away detail on the rare categories, which in high-cardinality data is most of them.</p><p><strong>Useful when:</strong> Cardinality is moderate and a few categories dominate.</p><h3>3.2 Frequency Encoding</h3><p><strong>Definition:</strong> Replace each category with how often it appears in the training data.</p><p><strong>How it works:</strong> Count each category in the training set and map the count to the category. Unseen test categories get 0.</p><p><strong>Example:</strong> If driver D0267 appears 52 times in training, every D0267 row becomes 52.</p><p><strong>Advantages:</strong> One column per feature, no target used, so no leakage.</p><p><strong>Limitations:</strong> Two different categories with the same count look identical. It only says how common a category is, not how it relates to the target.</p><p><strong>Useful when:</strong> Popularity itself is informative, e.g. fraud or demand patterns.</p><h3>3.3 Target (Mean) Encoding</h3><p><strong>Definition:</strong> Replace each category with the average target value for that category.</p><p><strong>How it works:</strong> Group the training data by category and take the mean of complained. Unseen categories fall back to the <strong>global mean</strong>, the overall complaint rate.</p><p><strong>Example:</strong> If a driver had 10 rides and 3 complaints, they become 0.30.</p><p><strong>Advantages:</strong> Packs the category’s relationship with the target into one number.</p><p><strong>Limitations:</strong> <strong>Target leakage.</strong> Each training row’s own label is included in its category’s mean, so the model sees a hint of the answer during training. Categories with few rows give noisy, extreme averages.</p><p><strong>Useful when:</strong> Cardinality is high and you have enough data per category, ideally combined with smoothing or cross-validation.</p><h3>3.4 Smoothed Target Encoding</h3><p><strong>Definition:</strong> Target encoding that blends each category’s mean with the global mean.</p><p><strong>How it works:</strong> The encoded value is a weighted mix: the more rows a category has, the more we trust its own mean. The knobs are min_samples_leaf and smoothing.</p><p><strong>Example:</strong> A driver with 1 ride and 1 complaint has a raw mean of 1.0, which is far too confident from one ride. Smoothing pulls it back toward the global ~0.16.</p><p><strong>Advantages:</strong> Protects against tiny categories producing extreme values.</p><p><strong>Limitations:</strong> Adds hyperparameters to tune, and does not fully remove train-time leakage.</p><p><strong>Useful when:</strong> Many categories are rare.</p><h3>3.5 Leave-One-Out Encoding</h3><p><strong>Definition:</strong> Target encoding where each training row’s own target is excluded from its category mean.</p><p><strong>How it works:</strong> For a training row, compute the category mean using all the <em>other</em> rows in that category. Small random noise is added to the training encoding so the model can’t reverse-engineer the label. Test rows use the normal training mean, without noise.</p><p><strong>Example:</strong> A driver has 3 complaints in 10 rides. For a row where the driver complained, the encoding is 2/9. For a row where they didn’t, it is 3/9.</p><p><strong>Advantages:</strong> Reduces self-target leakage compared with plain target encoding.</p><p><strong>Limitations:</strong> Rows in the same category get slightly different values depending on their label, which a flexible model can still exploit. Needs a noise setting (sigma).</p><p><strong>Useful when:</strong> You want target encoding with less leakage and have a reasonable number of rows per category.</p><h3>3.6 Weight of Evidence (WoE) Encoding</h3><p><strong>Definition:</strong> Replace each category with the log of the ratio of “good” to “bad” observations in that category.</p><p><strong>How it works:</strong> In binary classification, WoE = ln(share of all non-complaints in the category / share of all complaints in the category). Positive means the category leans toward no complaint, and negative means it leans toward complaints. The logarithm makes the scale symmetric around zero. The regularization parameter avoids dividing by zero.</p><p><strong>Example:</strong> If a zip holds 2% of all non-complaints but 4% of all complaints, WoE = ln(0.02/0.04) = about -0.69.</p><p><strong>Advantages:</strong> Comes from credit scoring, interpretable, and works well with linear-style models.</p><p><strong>Limitations:</strong> Still uses the target, so it can leak. Categories with a count of zero in one class need regularization.</p><p><strong>Useful when:</strong> Binary targets and interpretability matter.</p><h3>3.7 XGBoost Native Categorical Handling</h3><p><strong>Definition:</strong> Let XGBoost handle categorical columns directly, with no manual encoding.</p><p><strong>How it works:</strong> Columns are given the pandas category dtype and the model is created with enable_categorical=True. XGBoost then splits by <em>groups of categories</em> internally.</p><p><strong>Advantages:</strong> No separate encoding step, and no extra leakage from a manual encoder.</p><p><strong>Limitations:</strong> Needs the right dtype, supports a limited set of tree methods (I used tree_method=&quot;hist&quot;), and categories unseen in training become missing values.</p><p><strong>Useful when:</strong> You’re already using XGBoost.</p><h3>3.8 LightGBM Native Categorical Handling</h3><p><strong>Definition:</strong> LightGBM’s built-in support for categorical features.</p><p><strong>How it works:</strong> Columns have the pandas category dtype and are declared as categorical. LightGBM sorts categories by their gradient statistics and finds the best split among groups, instead of expanding into thousands of columns as One-Hot would.</p><p><strong>Advantages:</strong> Compact (one column per feature), fast, and no hand-built encoder.</p><p><strong>Limitations:</strong> Can overfit on rare categories, and has its own regularisation settings to tune.</p><p><strong>Useful when:</strong> You’re already using LightGBM and want a strong baseline.</p><h3>4. Experimental Methodology</h3><p>To make the comparison fair, I followed strict rules:</p><ul><li><strong>One train/test split</strong>, 80/20, stratified on complained (random_state=42), reused for all eight experiments.</li><li><strong>Encoders fitted on training data only.</strong> Test data was only transformed.</li><li><strong>vehicle_model One-Hot encoded the same way</strong> in every experiment.</li><li><strong>High-cardinality encoding applied only</strong> to driver_id, pickup_zip and dropoff_zip.</li><li><strong>Same LightGBM settings</strong> for experiments 1 to 6 (300 trees, learning rate 0.05, 31 leaves).</li><li><strong>Same metrics:</strong> ROC-AUC (primary) and Accuracy (secondary).</li></ul><p>The workflow:</p><pre>ride_hailing.csv<br>      |<br>      v<br>Drop index column -&gt; Train/Test split (once)<br>      |<br>      v<br>Base features: numeric columns + One-Hot(vehicle_model)<br>      |<br>      +--&gt; Encode driver_id, pickup_zip, dropoff_zip (8 variants, fit on TRAIN only)<br>                |<br>                v<br>          Train model (LightGBM / XGBoost)<br>                |<br>                v<br>      Evaluate on TEST: Accuracy + ROC-AUC<br>                |<br>                v<br>        Comparison table + chart</pre><h3>5. Implementation and Results</h3><p>A few snippets from my notebook.</p><p><strong>Capped One-Hot (Experiment 1):</strong></p><pre>ohe_capped = OneHotEncoder(<br>    max_categories=20,<br>    handle_unknown=&quot;infrequent_if_exist&quot;,<br>    sparse_output=False,<br>)<br>ohe_capped.fit(X_train[HIGH_CARD])</pre><p><strong>Frequency Encoding (Experiment 2):</strong></p><pre>freq_maps = {c: X_train[c].value_counts() for c in HIGH_CARD}<br>out[c + &quot;_freq&quot;] = X_[c].map(freq_maps[c]).fillna(0)   # unseen -&gt; 0</pre><p><strong>Target Encoding (Experiment 3):</strong></p><pre>global_mean = y_train.mean()<br>te_maps = {c: y_train.groupby(X_train[c]).mean() for c in HIGH_CARD}<br>out[c + &quot;_te&quot;] = X_[c].map(te_maps[c]).fillna(global_mean)   # unseen -&gt; global mean</pre><p><strong>Smoothed, Leave-One-Out and WoE</strong> used the category_encoders library:</p><pre>sm_enc  = ce.TargetEncoder(cols=HIGH_CARD, min_samples_leaf=20, smoothing=10)<br>loo_enc = ce.LeaveOneOutEncoder(cols=HIGH_CARD, sigma=0.05, random_state=42)<br>woe_enc = ce.WOEEncoder(cols=HIGH_CARD, regularization=1.0)</pre><p><strong>Native handling (Experiments 7 and 8):</strong> the category type is built from training categories, so unseen test values become missing instead of crashing.</p><pre>cat_types = {c: pd.CategoricalDtype(categories=sorted(X_train[c].unique()))<br>             for c in HIGH_CARD}<br>X_[c] = X_[c].astype(cat_types[c])</pre><pre>XGBClassifier(tree_method=&quot;hist&quot;, enable_categorical=True, max_cat_to_onehot=1)<br>lgb.LGBMClassifier(**LGB_PARAMS).fit(Xtr, y_train, categorical_feature=HIGH_CARD)</pre><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7r6lHUwMPe1Ka-mES_ivkQ.png" /></figure><p><strong>Accuracy vs ROC-AUC bar chart</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KCpO6PlAiygFkB0uKSYf2g.png" /></figure><h3>6. Results and Discussion</h3><p><strong>Best by ROC-AUC: Leave-One-Out (0.7077).</strong> It separated complainers from non-complainers better than any other technique.</p><p><strong>Best by Accuracy: Frequency (0.8393)</strong>, which is a trap. About 84% of rides have no complaint, so a model that always predicts “no complaint” gets about 0.84 without learning anything. Frequency’s AUC of 0.5619 shows it is barely better than guessing. Capped One-Hot has the same problem: Accuracy of 0.8387 and AUC of just 0.5405. So <strong>the Accuracy winner and the ROC-AUC winner were different</strong>, and picking on Accuracy alone would have chosen a nearly useless model.</p><p><strong>Poor performers:</strong> Capped One-Hot and Frequency. One-Hot kept only the 19 most common values out of 600 drivers and 1,500 zips, and lumped the rest together, discarding most of the information. Frequency says how <em>common</em> a driver or zip is, but not how <em>complaint-prone</em> it is.</p><p><strong>Target vs Smoothed Target:</strong> Smoothing helped: AUC rose from 0.6608 to 0.6888. Rare drivers and zips give extreme raw averages (a driver with one ride and one complaint scores 1.0). Smoothing pulls those toward the global mean so the model trusts them less.</p><p><strong>Leave-One-Out vs Target Encoding:</strong> LOO scored 0.7077 against 0.6608 for plain target encoding. In plain target encoding, each training row’s label is part of its own category mean, which leaks the answer. LOO removes that and adds noise, which made the training features more honest.</p><p><strong>Native vs manual encoding:</strong> Native LightGBM (0.6926) and native XGBoost (0.6866) performed strongly with zero manual encoding work. They beat plain Target and WoE, and were about level with Smoothed Target. Only Leave-One-Out finished clearly ahead. The differences among Smoothed Target and the two native methods are small, and with a single split I would call them comparable.</p><h3>7. Challenges and Learnings</h3><ul><li><strong>High cardinality:</strong> One-Hot grew the feature set from 12 to 69 even after capping. Uncapped it would have been over 3,600.</li><li><strong>Target leakage:</strong> The biggest lesson. Plain target encoding looks innocent, but the model can see the answer. I avoided leakage by fitting every encoder on training data only, and by using Leave-One-Out and smoothing where they apply.</li><li><strong>Unseen categories:</strong> Each technique needs a plan: frequency gives 0, target-based methods use the global mean, One-Hot sends them to the infrequent bucket, and native methods treat them as missing.</li><li><strong>Accuracy vs ROC-AUC:</strong> I almost misjudged the results because of class imbalance. Always check the baseline.</li><li><strong>Practical issues:</strong> Matching dtypes for native handling, and cleaning column names for LightGBM.</li></ul><h3>8. Conclusion</h3><p>For this ride-hailing dataset, <strong>Leave-One-Out encoding performed best</strong> with ROC-AUC 0.7077, and Smoothed Target and the two native categorical methods followed closely. Capped One-Hot and Frequency encoding looked fine on Accuracy but were close to useless at ranking complaints.</p><p>The real takeaway isn’t that Leave-One-Out always wins. It’s that <strong>encoding choice should come from experiments, not assumptions</strong>, evaluated with the right metric, using an encoder fitted only on training data. A different dataset, with different cardinality, signal strength and class balance, could easily rank these methods differently.</p><h3>9. References</h3><ul><li>scikit-learn: OneHotEncoder documentation (scikit-learn.org)</li><li>category_encoders library documentation (contrib.scikit-learn.org/category_encoders)</li><li>LightGBM documentation: Features, Categorical Feature Support (lightgbm.readthedocs.io)</li><li>XGBoost documentation: Categorical Data (xgboost.readthedocs.io)</li><li>Micci-Barreca, D. (2001). <em>A Preprocessing Scheme for High-Cardinality Categorical Attributes in Classification and Prediction Problems.</em> ACM SIGKDD Explorations.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bc2c2405404" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Rules to Learning: A Revision Guide to Machine Learning Fundamentals]]></title>
            <link>https://medium.com/@abhishekareddy1234/from-rules-to-learning-a-revision-guide-to-machine-learning-fundamentals-8abf640a0c10?source=rss-f2e9f4dbcfd0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8abf640a0c10</guid>
            <dc:creator><![CDATA[Abhishekar Reddy]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 11:39:01 GMT</pubDate>
            <atom:updated>2026-11-03T11:39:01.286Z</atom:updated>
            <content:encoded><![CDATA[<p>A revision guide on why ML exists, how models learn, the main learning paradigms, and where ML is used in the real world.</p><h4>Introduction :</h4><p>When I started learning Machine Learning, I could recite “ML is a subset of AI that learns from data” without really understanding it. This article is my attempt to go deeper: why ML was needed, how models actually learn, how the learning styles differ, and where all of it shows up in real life. I’m writing it as a revision guide for future projects and interviews. What you’ll learn</p><ol><li>The evolution of ML: from rules to data-driven learning</li></ol><p>2. How ML models learn: parametric vs non-parametric, generalization vs memorization, and two classic animal experiments</p><p>3. Types of ML: supervised, unsupervised, reinforcement, self-supervised</p><p>4. Types of problems: classification vs regression</p><p>5. ML in the real world</p><h4>Section 1: The Evolution of Machine Learning</h4><p><em>1.1 Rule-Based Programming and Its Limitations</em></p><p>Traditional programming works like a recipe. A human studies the problem, writes the rules, and the computer follows them exactly.</p><p><strong>Rules + Data → Program → Answers</strong></p><p>Suppose we want to filter spam emails. A rule-based approach might look like this:</p><pre>def is_spam(email_text):<br>spam_words = [&quot;lottery&quot;, &quot;free money&quot;, &quot;winner&quot;, &quot;click here&quot;]<br>text = email_text.lower()<br>for word in spam_words:<br>if word in text:<br>return True<br>return False<br>print(is_spam(&quot;You are a WINNER! Click here&quot;))   # True<br>print(is_spam(&quot;Cl1ck h3re to cl@im your pr1ze&quot;)) # False (spammer changed <br>spelling)</pre><p>This works until it doesn’t. Spammers change spelling, languages differ, and new tricks appear every week. The rule list grows forever and still misses things</p><h4>Key limitations of rule-based systems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wS0jUdItxKvQe5aEy6lnOg.png" /></figure><p><em>1.2 The Evolution and Need for Machine Learning</em></p><p>The idea of machines that learn is not new. In 1959, Arthur Samuel described machine learning as giving computers the ability to learn without being explicitly programmed, after building a checkers program that improved by playing itself. Later, Tom Mitchell (1997) gave a more precise definition:</p><p>A program learns from experience E with respect to task T and performance measure P, if its performance on T, as measured by P, improves with E</p><p>For spam filtering: T = classify emails, E = emails labeled spam/not spam, P= percentage classified correctly.</p><p>ML flips the traditional recipe:</p><p><strong>Data + Answers → ML Algorithm → Rules (the model)</strong></p><p>Instead of writing the rules, we show examples and let the algorithm find the rules itself.</p><h4>A rough timeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wofKdUwGt1VDnUQTErNDgQ.png" /></figure><blockquote>Why ML became necessary</blockquote><ul><li><strong>Data explosion:</strong> we produce huge amounts of data (clicks, sensors, transactions, images) that no human can turn into rules by hand.</li><li><strong>Complex patterns:</strong> speech, vision, and language can’t be written as if-else statements.</li><li><strong>Personalization at scale: </strong>millions of users need different recommendations.</li><li><strong>Adaptability: </strong>models can be retrained as the world changes.</li><li><strong>Better compute:</strong> GPUs and cloud platforms made training practical.</li></ul><p><em>1.3 The Role of Data Analysis in Building ML Models</em></p><p>A model is only as good as the data it learns from. “Garbage in, garbage out” is the most reliable law in ML. That is why data analysis comes before modeling.</p><h4><strong>A typical ML workflow</strong></h4><p><em>Problem definition → Data collection → Data cleaning → Exploratory Data Analysis (EDA) → Feature engineering → Model training → Evaluation → Deployment → Monitoring</em></p><h4>What data analysis gives us</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3jXyJP2l-jyHVHqx7neWSA.png" /></figure><p>A small example:</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>df = pd.read_csv(&quot;houses.csv&quot;)<br>print(df.info())                 <br>print(df.describe())             <br>print(df.isna().sum())           <br># types and non-null counts<br># summary statistics<br># missing values per column<br>print(df.corr(numeric_only=True)[&quot;price&quot;].sort_values())  # what relates to <br>price?<br>df = df.dropna()<br>X = df.drop(&quot;price&quot;, axis=1)<br>y = df[&quot;price&quot;]<br>X_train, X_test, y_train, y_test = train_test_split(<br>X, y, test_size=0.2, random_state=42<br>)</pre><p><strong>Why this matters:</strong> bad data leads to bad models. A model trained on biased data produces biased predictions. EDA is where you catch problems early, such as a column that leaks the answer, a target that is wildly imbalanced, or features on very different scales.</p><h4>Section 2: Understanding How ML Models Learn</h4><p><em>2.1 Parametric and Non-Parametric Models</em></p><p>This is a question about how much structure a model assumes before seeing the data.</p><p>Parametric models assume a fixed form (a formula) with a fixed number of parameters. Training means finding the best values for those parameters. After training, the data can be thrown away; the parameters are the knowledge.</p><ul><li>Example: Linear Regression assumes <strong>y = w1*x1 + w2*x2 + … + b</strong>. Whether you have 100 rows or 10 million, there are still the same number of weights.</li><li>Others: Logistic Regression, Naive Bayes, a neural network with fixed architecture.</li></ul><p><strong>Non-parametric models</strong> don’t fix the model’s shape in advance. Their complexity grows with the data. They often need to keep the training data (or a structure built from it) around to make predictions.</p><ul><li>Example: k-Nearest Neighbors predicts by looking at the closest stored examples.</li><li>Others: Decision Trees, Random Forests, kernel SVMs, Gaussian Processes.</li></ul><p>(Note: “non-parametric” doesn’t mean “no parameters”. It means the number of parameters isn’t fixed in advance.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HhN63fj7Yamh9XSWltJ-xw.png" /></figure><pre>from sklearn.linear_model import LinearRegression<br>from sklearn.neighbors import KNeighborsRegressor<br>parametric = LinearRegression().fit(X_train, y_train)          # learns a few <br>weights<br>nonparametric = KNeighborsRegressor(n_neighbors=5).fit(X_train, y_train)  # <br>stores the data<br>print(&quot;Linear R²:&quot;, parametric.score(X_test, y_test))<br>print(&quot;kNN R²   :&quot;, nonparametric.score(X_test, y_test))</pre><p><strong>Rule of thumb: </strong>if you have little data and a reasonable idea of the relationship, parametric is a good start. If you have lots of data and a complex, unknown relationship, non parametric methods give more freedom.</p><p><em>2.2 Generalization and Memorization</em></p><p>The goal of ML is not to do well on the data it has seen. It is to do well on data it has never seen. That ability is called generalization.</p><p><strong>Analogy:</strong> two students prepare for an exam.</p><ul><li>Student A memorizes last year’s question paper word for word.</li><li>Student B understands the concepts.</li></ul><p>If the exam repeats last year’s questions, A does great. If the questions are new, A fails while B succeeds. A has memorized; B has generalized. In ML terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W0PUFwBQAKgZR7_2aHv_ug.png" /></figure><pre>from sklearn.tree import DecisionTreeClassifier<br>Good generalization<br>Overfitting (high<br>variance)<br># An unrestricted tree can memorize the training set<br>overfit = DecisionTreeClassifier(max_depth=None).fit(X_train, y_train)<br>print(&quot;Train:&quot;, overfit.score(X_train, y_train))  # often ~1.00<br>print(&quot;Test :&quot;, overfit.score(X_test, y_test))    <br># noticeably lower<br># Limiting depth encourages generalization<br>better = DecisionTreeClassifier(max_depth=4).fit(X_train, y_train)<br>print(&quot;Train:&quot;, better.score(X_train, y_train))<br>print(&quot;Test :&quot;, better.score(X_test, y_test))</pre><p>A big gap between training and test scores is the classic sign of overfitting.</p><p><strong>How to improve generalization</strong></p><ul><li>Collect more (and more varied) data</li><li>Hold out a test set and use cross-validation</li><li>Use regularization (L1/L2, dropout)</li><li>Simplify the model, prune trees, or stop training early</li><li>Use data augmentation</li><li>Remove noisy or leaky features</li></ul><p>This is the<strong> bias-variance trade-off:</strong> too simple a model has high bias (misses real patterns), too complex a model has high variance (chases noise). Good ML is finding the balance</p><p><em>2.3 The Rat-and-Food and Pigeon-Superstition Examples</em></p><p>(Note: I’m describing these as they relate to the classic behavioral experiments. Adjust the details to match your class notes if your instructor framed them slightly differently.)</p><p>Both examples come from behavioral psychology, and they teach opposite sides of the same lesson about learning from experience.</p><p><strong>The rat-and-food example: learning a real cause and effect</strong></p><p>In B.F. Skinner’s operant conditioning experiments, a hungry rat is placed in a box with a lever. Initially it wanders and sometimes presses the lever by accident. Whenever a press delivers a food pellet, the rat slowly learns: pressing the lever → food. Over time the rat presses deliberately.</p><p>What is happening in ML terms:</p><ul><li><strong>Agent:</strong> the rat</li><li><strong>Action: </strong>press or don’t press the lever</li><li><strong>Reward:</strong> food</li><li><strong>Learned policy:</strong> press the lever when hungry</li></ul><p>The link between action and reward is real and consistent, so the learned behavior is genuinely useful. This is the intuition behind reinforcement learning: behavior followed by reward gets repeated</p><p><strong>The pigeon-superstition example: learning a false pattern</strong></p><p>In 1948, Skinner ran a different experiment. Hungry pigeons were given food at regular intervals <strong>regardless of what they did.</strong> Yet many pigeons developed odd rituals: turning counterclockwise, head-tossing, swinging their heads. Whatever the bird happened to be doing when food arrived got “credited” for the food, so it repeated the action, and by chance was often rewarded again, which strengthened the belief. Skinner compared this to human superstitions.</p><p>There was no real cause and effect. The pigeons learned a <strong>spurious correlation.</strong></p><p><strong>Why this matters for ML</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AAhfqBrJrqL8HTIzumHxGA.png" /></figure><p>Pigeon-style mistakes happen in real ML. A model trained to detect pneumonia in X-rays once relied partly on hospital-specific markers in the image rather than the lungs. A “wolf vs husky” classifier learned to detect snow in the background. The model found a pattern that correlated with the label in training but had nothing to do with the real cause, and it failed in the real world.</p><p><strong>Takeaway:</strong> a model can be confidently wrong when it learns coincidences. More diverse data, proper validation, and domain sense are the protection against becoming a superstitious pigeon.</p><h4>Section 3: Types of Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJ4xHs75hx0vzF0e6oj5Og.png" /></figure><p><em>3.1 Supervised Learning</em></p><p><strong>Basic idea:</strong> learn from examples that come with the correct answer (labels), like a student studying with an answer key.</p><p><strong>How it works:</strong></p><ol><li>Collect data with inputs (X) and known outputs (y)</li></ol><p>2. The model predicts, compares its prediction to the true label, and measures the error using a loss function</p><p>3. It adjusts its parameters to reduce the error, and repeats</p><p>4. Finally, it predicts on new, unlabeled inputs</p><pre>from sklearn.datasets import load_iris<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.model_selection import train_test_split<br>X, y = load_iris(return_X_y=True)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, <br>random_state=42)<br>model = RandomForestClassifier(random_state=42).fit(X_train, y_train)<br>print(&quot;Accuracy:&quot;, model.score(X_test, y_test))</pre><p><strong>Real-world example:</strong> a bank trains a model on past loan applications labeled “repaid” or “defaulted” to predict whether a new applicant is risky. <strong>Common algorithms:</strong> Linear/Logistic Regression, Decision Trees, Random Forest, SVM, kNN, Neural Networks, Gradient Boosting.</p><p><em>3.2 Unsupervised Learning</em></p><p><strong>Basic idea:</strong> there are no answers provided. The model explores the data to find hidden structure on its own, like sorting a pile of unlabeled photos into groups without being told the categories.</p><p><strong>How it works:</strong> the algorithm looks at similarity, density, or variance in the data to find groups, simplify features, or spot oddities.</p><pre>from sklearn.cluster import KMeans<br>kmeans = KMeans(n_clusters=3, random_state=42, n_init=10).fit(X)<br>print(kmeans.labels_[:10])   # which cluster each customer/sample belongs to</pre><p><strong>Main tasks</strong></p><ul><li><strong>Clustering:</strong> group similar items (K-Means, DBSCAN, Hierarchical)</li><li><strong>Dimensionality reduction:</strong> compress many features into few (PCA, t-SNE, UMAP)</li><li><strong>Anomaly detection:</strong> find unusual points (Isolation Forest)</li><li><strong>Association rules:</strong> find items that occur together (Apriori)</li></ul><p><strong>Real-world example:</strong> a retailer clusters customers by buying behavior into groups such as “bargain hunters”, “loyal regulars”, and “big spenders”, then designs different marketing for each, without anyone labeling the groups beforehand.</p><p><em>3.3 Reinforcement Learning</em></p><p><strong>Basic idea</strong>: an<strong> agent</strong> learns by interacting with an <strong>environment</strong>, taking actions and receiving<strong> rewards</strong> or penalties. It’s learning by trial and error, like training a dog with treats, or like the rat in the box.</p><p><strong>How it works:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OdXV402iR6jIvBuEXMa5Zg.png" /></figure><ol><li>The agent observes the current state</li></ol><p>2. It picks an action</p><p>3. The environment returns a new state and a reward</p><p>4. The agent updates its strategy (policy) to maximize total future reward</p><p>A key challenge is <strong>exploration vs exploitation:</strong> try new things, or stick with what already works?</p><pre># Core idea of Q-learning update (simplified)<br># Q[state, action] += alpha * (reward + gamma * max(Q[next_state]) - Q[state, <br>action])</pre><p><strong>Real-world examples:</strong> DeepMind’s AlphaGo learning to beat top Go players, robots learning to walk or grasp objects, dynamic pricing, and traffic-signal control. Reinforcement learning from human feedback (RLHF) is also used to fine-tune chatbots.</p><p><em>3.4 Self-Supervised Learning</em></p><p><strong>Basic idea:</strong> the model creates its <strong>own labels from the data itself.</strong> No humans annotate anything, yet the model gets a supervised-style learning signal. It sits between supervised and unsupervised learning.</p><p><strong>How it works:</strong> hide part of the data and ask the model to predict it.</p><ul><li><strong>Language:</strong> hide the next word (or a masked word) and predict it. “The cat sat on the ___” → “mat”.</li><li><strong>Vision:</strong> mask a patch of an image and reconstruct it, or learn that two crops of the same image should be similar (contrastive learning).</li><li><strong>Audio:</strong> predict the missing piece of a sound clip.</li></ul><p>This lets models learn from huge amounts of raw, unlabeled data. The learned knowledge (representations) is then fine-tuned on a small labeled dataset for a specific task.</p><p><strong>Real-world example:</strong> large language models like GPT and BERT are pretrained by self supervised learning on massive text, then adapted for chat, translation, or summarization.</p><p><strong>Comparison at a Glance</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KpW-5WcdqphCficU8WkHEw.png" /></figure><h4>Section 4: Types of ML Problems</h4><p>Within supervised learning, almost every problem is either <strong>classification or regression.</strong></p><p>The deciding question is simple: <strong>what kind of thing are you predicting?</strong></p><p><em>4.1 Classification</em></p><p>Predicts a <strong>category (discrete label).</strong></p><ul><li><strong>Binary: </strong>two classes (spam / not spam; fraud / genuine)</li><li><strong>Multi-class:</strong> many classes, one per item (digit 0–9; type of flower)</li><li><strong>Multi-label:</strong> multiple labels at once (a movie tagged “comedy” and “romance”)</li></ul><p><strong>Common algorithms:</strong> Logistic Regression, Decision Tree, Random Forest, SVM, kNN, Naive Bayes, Neural Networks.</p><p><strong>Metrics:</strong> accuracy, precision, recall, F1-score, ROC-AUC, confusion matrix.</p><pre>from sklearn.linear_model import LogisticRegression<br>from sklearn.metrics import classification_report<br>clf = LogisticRegression(max_iter=200).fit(X_train, y_train)<br>print(classification_report(y_test, clf.predict(X_test)))</pre><p><strong>Examples:</strong> email spam detection, disease diagnosis (positive/negative), loan default prediction, handwritten digit recognition, sentiment analysis.</p><p><em>4.2 Regression</em></p><p>Predicts a <strong>continuous numeric value.</strong></p><p><strong>Common algorithms:</strong> Linear Regression, Ridge/Lasso, Decision Tree Regressor, Random Forest Regressor, Gradient Boosting, SVR.</p><p><strong>Metrics:</strong> MAE, MSE, RMSE, R².</p><pre>from sklearn.linear_model import LinearRegression<br>from sklearn.metrics import mean_absolute_error, r2_score<br>reg = LinearRegression().fit(X_train, y_train)<br>pred = reg.predict(X_test)<br>print(&quot;MAE:&quot;, mean_absolute_error(y_test, pred))<br>print(&quot;R² :&quot;, r2_score(y_test, pred))</pre><p><strong>Examples:</strong> house price prediction, forecasting sales, estimating delivery time, predicting temperature, estimating a patient’s blood sugar level.</p><p><em>4.3 Classification vs Regression</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sOPB9VIjpxlQiQDEZlsizw.png" /></figure><p><strong>Tricky cases to remember</strong></p><ul><li>Predicting a rating from 1–5 can be treated as classification (5 classes) or regression (a number). The choice depends on whether the order and distance between values matter.</li><li>Logistic Regression has “regression” in its name but is a classification algorithm. It outputs a probability, then a class.</li><li>A regression output can be turned into classification by thresholding (e.g., predicted risk score &gt; 0.5 → “high risk”)</li></ul><h4>Section 5: Machine Learning in the Real World</h4><p>ML is no longer a lab experiment. It is embedded in everyday products. Here are the main domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C_M8EwWCPVr4q0_7KPuz6Q.png" /></figure><p><strong>How recommendation systems work in short:</strong> they predict what you will like using (a) what similar users liked (collaborative filtering), (b) the properties of items you already enjoyed (content-based filtering), or © hybrids of both with deep learning.</p><p><strong>A note of caution: </strong>ML in the real world also raises serious questions. Models can inherit bias from data, make opaque decisions, and fail on situations unlike their training data (remember the pigeon). In healthcare and finance especially, human oversight, fairness checks, and explainability matter as much as accuracy.</p><h4>Quick Revision Cheat Sheet</h4><ul><li><strong>Rule-based programming:</strong> humans write rules; breaks on complexity and change.</li><li><strong>ML:</strong> data + answers produce rules; scales to messy, complex problems.</li><li><strong>Data analysis (EDA) comes first:</strong> clean data, understand patterns, split train/test.</li><li><strong>Parametric:</strong> fixed number of parameters, strong assumptions, fast (Linear Regression). <strong>Non-parametric:</strong> flexible, grows with data (kNN, trees).</li><li><strong>Generalization:</strong> performing well on unseen data is the real goal. <strong>Memorization (overfitting)</strong> = great on training, poor on test.</li><li><strong>Rat and food: </strong>real cause-effect learned through reward (reinforcement idea). <strong>Pigeon superstition:</strong> coincidence mistaken for cause (spurious correlation, overfitting).</li><li><strong>Supervised:</strong> labeled data. <strong>Unsupervised:</strong> find structure. <strong>Reinforcement:</strong> learn via rewards. <strong>Self-supervised:</strong> make labels from the data itself.</li><li><strong>Classification:</strong> predict a category. <strong>Regression:</strong> predict a number.</li><li>ML is used across healthcare, finance, e-commerce, education, agriculture, and recommendations.</li></ul><p><strong>Conclusion :</strong></p><p>Machine Learning exists because writing rules by hand doesn’t scale to the messy, changing, data-rich world we live in. The core skill isn’t memorizing algorithms; it’s understanding how a model learns, making sure it learns real patterns instead of coincidences, picking the right learning paradigm and problem type, and checking that it works on data it has never seen. If you keep the rat and the pigeon in mind, you’ll remember the central lesson: learn the signal, not the noise.</p><p>Thank you for reading! If this helped your revision, feel free to share it and let me know what to cover next.</p><p><strong>References :</strong></p><ol><li>Samuel, A. L. (1959). Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development.</li></ol><p>2. Mitchell, T. M. (1997). Machine Learning. McGraw-Hill.</p><p>3. Skinner, B. F. (1948). “Superstition” in the Pigeon. Journal of Experimental Psychology, 38(2), 168–172.</p><p>4. Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow. O’Reilly.</p><p>5. Scikit-learn documentation: <a href="https://scikit-learn.org/stable/">https://scikit-learn.org/stable/</a></p><p>6. Sutton, R. S., &amp; Barto, A. G. Reinforcement Learning: An Introduction. MIT Press.</p><p>7. Ribeiro, M. T., Singh, S., &amp; Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. (wolf vs husky example)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8abf640a0c10" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>