<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by AlgoStalk on Medium]]></title>
        <description><![CDATA[Stories by AlgoStalk on Medium]]></description>
        <link>https://medium.com/@algostalk?source=rss-06c43f99303c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*VxlPKbVgAIpQISUi</url>
            <title>Stories by AlgoStalk on Medium</title>
            <link>https://medium.com/@algostalk?source=rss-06c43f99303c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:55:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@algostalk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Model Was Running Fine. The Problem Was What Its Defaults Meant.]]></title>
            <link>https://medium.com/@algostalk/the-model-was-running-fine-the-problem-was-what-its-defaults-meant-a101a1c85859?source=rss-06c43f99303c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a101a1c85859</guid>
            <dc:creator><![CDATA[AlgoStalk]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 01:35:13 GMT</pubDate>
            <atom:updated>2026-10-30T01:35:13.100Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A reproducible AdaBoostRegressor case study on alert quality, loss functions, and why an AI-generated baseline is not the end of the investigation.</em></p><blockquote><strong>Before you continue:</strong> for a visual explanation of how AdaBoost works, watch the companion video on the AlgoStalk YouTube channel. This article does not repeat that fundamentals lesson. It investigates one operational consequence of AdaBoostRegressor’s loss setting.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fp_aNBtCLT0o%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dp_aNBtCLT0o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fp_aNBtCLT0o%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/484602a3ffb0cc971905877b26a688a3/href">https://medium.com/media/484602a3ffb0cc971905877b26a688a3/href</a></iframe><h3>The hard part is not finding a spike</h3><p>The hard part is deciding which spikes deserve human attention.</p><p>An AI agent can instantiate an estimator, run a notebook, and report a reasonable aggregate score in seconds. That is useful. It is also incomplete. A model can run without errors while creating the wrong alerts: asking people to investigate known, legitimate behavior while offering no extra clarity on genuinely unexplained behavior.</p><p>I built a small, fully reproducible traffic experiment to make that distinction concrete. The result is not a claim about any production system or security incident. The data are synthetic by design, because the real site data and implementation cannot be published. The values, charts, calculations, and outputs below are from the actual deterministic synthetic experiment in this project — not invented examples after the fact.</p><p>The operational objective is narrow:</p><blockquote><em>Estimate expected hourly traffic from known inputs. Route unusually large residuals to human review. Do not treat a residual as an attack classification.</em></blockquote><p>The experiment contains a rare but legitimate campaign pattern and a separate simulated unexplained spike. The question is not whether a model can see a high number. It is whether it learns the known pattern well enough that human attention remains available for the unknown one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8IeHGJgcR7smdfZDtRSWkw.png" /><figcaption><em>Figure 1. Running a model is not the same as understanding its behaviour.</em></figcaption></figure><h3>The data: a controlled hourly traffic experiment</h3><p>The dataset has 4,320 hourly observations: 180 days beginning 1 January 2025. It includes daily seasonality, weekday effects, Gaussian noise, and recurring two-day marketing events every 21 days.</p><p>Most marketing events add a modest lift. A rarer interaction adds a much larger but legitimate lift: the campaign is active, the time is 8pm, and the event falls on its scheduled weekend. Those inputs are available to the model. This matters: the model is not expected to infer an unrecorded cause.</p><p>In the held-out test period, I add one similarly sized one-hour simulated surge with no explanatory feature. It is deliberately separate from the campaign mechanism. The model is not given an “attack” column or a hidden proxy for it.</p><p>Item: Configuration</p><p>Observations: 4,320 hourly rows / 180 days</p><p>Target: Hourly website visits</p><p>Features: Hour-of-day sine/cosine, day-of-week sine/cosine, marketing-event flag, weekend flag</p><p>Legitimate rare pattern: Campaign event at 8pm on its event weekend</p><p>Simulated unexplained pattern: One held-out hourly spike, with no explanatory feature</p><p>Split: Chronological: 60% train / 20% validation / 20%</p><p>testSeed: 20261023</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U0Lxy68J_zs8BRDywNiHrw.png" /><figcaption><em>Figure 2. The full 180-day synthetic experiment shown as daily totals. Blue points identify rare known campaign hours; orange identifies the held-out unexplained spike. Daily aggregation deliberately makes a one-hour spike less prominent; Figure 5 shows the same observations at hourly resolution.</em></figcaption></figure><h3>The fixed model comparison</h3><p>I used scikit-learn 1.9.0’s AdaBoostRegressor with a fixed shallow base tree:</p><pre>AdaBoostRegressor(<br>    estimator=DecisionTreeRegressor(max_depth=3, random_state=20261023),<br>    n_estimators=20,<br>    learning_rate=0.05,<br>    loss=loss,                 # linear, square, or exponential<br>    random_state=20261023,<br>)</pre><p>Everything except loss stayed fixed: data, features, chronological split, base-tree depth, estimator count, learning rate, and random seed. This isolates the decision under investigation. The validation slice confirmed the pattern; the final numbers below come from the untouched test period.</p><h3>Why loss matters inside AdaBoostRegressor</h3><p>AdaBoostRegressor does not simply fit a larger tree when it makes a mistake. It fits shallow regressors sequentially. After each round, it changes the distribution from which the next learner is trained.</p><p>For a given round, scikit-learn takes the absolute residual and normalises it by that round’s largest residual. Call the result e, where 0 &lt;= e &lt;= 1. The relevant transformations are:</p><pre>linear: L(e) = e<br>square: L(e) = e^2</pre><p>It then computes weighted round error E, defines beta = E / (1 - E), and updates the sample weight using:</p><pre>weight *= beta ** ((1 - L(e)) * learning_rate)</pre><p>Because beta is below one, a higher transformed loss means that example is downweighted less for the next round. The point is subtle: square loss does not simply make every error “more important.” It downplays middle-sized normalised errors relative to errors near the maximum.</p><p>At e = 0.6, linear gives 0.60 and square gives 0.36. At e = 0.9, they give 0.90 and 0.81. The setting changes which mistakes later trees are built to address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sj2vTxNjN2dEnDSVLZXo-w.png" /><figcaption><em>Figure 3. In AdaBoostRegressor, </em><em>loss is a reweighting policy, not the final prediction metric.</em></figcaption></figure><h3>The result that an aggregate report can hide</h3><p>The default loss=&#39;linear&#39; looked serviceable on ordinary test traffic: MAE 59.52. A quick model report could stop there.</p><p>That would miss the operational failure. On the rare, known campaign slice, linear loss produced MAE 700.65. The model would have treated expected business activity as a major deviation from expectation.</p><p>Square loss changed that picture. Its ordinary-traffic MAE was only slightly higher at 60.81, while rare-campaign MAE fell to 41.31. Exponential loss did not help that rare slice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DdKId82289K4qrS4cqJAdw.png" /></figure><p>The last row is important. Changing loss did not “detect an attack.” The unexplained spike remains a large residual under every setting because no model input explains it. It is a review signal, not proof of a cause.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mrZN1Oc2W1vcDO8T7hQWfg.png" /><figcaption><em>Figure 4. Similar ordinary-traffic performance concealed a dramatically different outcome on the rare legitimate pattern.</em></figcaption></figure><h3>What the two spikes show</h3><p>The final visual is the practical result in miniature.</p><p>On the left, the observed campaign spike is legitimate and feature-defined. Linear loss materially underpredicts it; square loss predicts it closely. A residual-based alert system built on the linear model could waste human attention on an event the available inputs should make predictable.</p><p>On the right, the observed spike is intentionally unexplained. Neither model has a causal input that would allow it to forecast the lift. Both predictions remain near expected traffic and both residuals remain large. That is the behaviour wanted from this controlled setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObNa_hPFDXDl-f19_zxW6Q.png" /><figcaption><em>Figure 5. Square loss learns the rare known campaign pattern without normalising the unexplained spike.</em></figcaption></figure><h3>What an AI agent can do — and what it cannot decide for you</h3><p>An AI agent can generate the code, execute the default model, compare options, and surface the metric table. Those are meaningful capabilities. But the agent does not automatically know that false alarms during rare campaigns are costly, or that an unexplained residual should lead to review rather than a claim of attack. Those are problem-definition and decision-design choices.</p><p>This is where experienced data-science judgement matters: inspect defaults, understand the training mechanism, ask which slice carries operational cost, and keep a held-out test period intact. “The model ran” and “the model supports the decision” are different statements.</p><p>If you want the foundational walkthrough before applying this idea to a case study, watch the AdaBoost companion video on the <a href="https://www.youtube.com/@AlgoStalk">AlgoStalk YouTube channel</a>. The video explains the mechanics; this article shows why those mechanics matter when a model enters an alerting workflow.</p><h3>Limits of this demonstration</h3><p>This is a controlled synthetic stress test. It deliberately creates a rare interaction for a shallow ensemble to learn. The result does not establish that square loss will outperform linear loss on another site, another target, or a production system.</p><p>The simulated spike is not an attack label. In practice, a high residual could reflect an unlogged campaign, instrumentation error, a release, automated traffic, or a genuine incident. It should trigger investigation, not a causal conclusion.</p><p>The correct lesson is narrower and more useful: defaults are starting points. In AdaBoostRegressor, loss defines part of the error-reweighting policy. If the cost of errors is asymmetric or concentrated in a rare operating scenario, inspect that behaviour directly rather than trusting a single aggregate number.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a101a1c85859" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Wisdom of the Forest: What Random Forest Regressor Taught Me About Collective Intelligence]]></title>
            <link>https://medium.com/@algostalk/the-wisdom-of-the-forest-what-random-forest-regressor-taught-me-about-collective-intelligence-9e835fffa6a5?source=rss-06c43f99303c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e835fffa6a5</guid>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[AlgoStalk]]></dc:creator>
            <pubDate>Sun, 26 Oct 2025 03:56:01 GMT</pubDate>
            <atom:updated>2025-11-07T00:12:20.482Z</atom:updated>
            <content:encoded><![CDATA[<p>There’s something fascinating about nature — no single tree can make a forest, and no forest thrives if every tree grows the same way.</p><p>The same wisdom hides inside one of the most powerful algorithms in machine learning — the <strong>Random Forest Regressor</strong>.</p><p>Before we step into this forest, let’s revisit our old friend, the <strong>Decision Tree</strong>. In my earlier story, I explained how a single tree makes predictions by following simple clues — one branch at a time, one decision after another. It’s a beautiful idea, but also a fragile one. A single tree can easily get biased by noisy data, just like one detective can get misled by a tricky clue.</p><p>That’s where the Random Forest steps in — it doesn’t rely on just one detective. It builds a <em>team</em> of them. Each tree works on its own clues, and together, they reach a wiser, more reliable verdict.</p><p>If you’d like a quick refresher, you can revisit my earlier piece on <a href="https://medium.com/@algostalk/decision-tree-regressor-the-unsung-foundation-for-mastering-machine-learning-using-elementary-a036911f64cf"><strong>Decision Tree Regressor: The Unsung Foundation for Mastering Machine Learning</strong></a> — it’ll help this forest make even more sense.</p><p>🎥 <strong>Prefer visual learning?</strong><br> You can watch my YouTube videos for an animated walkthrough:</p><ul><li><a href="https://youtu.be/SQbXSpOMewc">Decision Tree Regressor Explained</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSQbXSpOMewc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSQbXSpOMewc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSQbXSpOMewc%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/805ce613c675d964581d59cbf72ad8a9/href">https://medium.com/media/805ce613c675d964581d59cbf72ad8a9/href</a></iframe><ul><li><a href="https://youtu.be/WqKgrz0-NxI">Random Forest Regressor: How the Trees Work Together</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWqKgrz0-NxI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWqKgrz0-NxI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWqKgrz0-NxI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecfc66354b6aae1f5e36f5a2c0b4b532/href">https://medium.com/media/ecfc66354b6aae1f5e36f5a2c0b4b532/href</a></iframe><h3>🍋 Why Do We Need a Random Forest?</h3><p>Imagine you’re running a small lemonade stand in your neighborhood. Some days, sales soar. Other days, barely anyone stops by. Naturally, you start wondering — <em>what’s causing this pattern?</em></p><p>In our earlier case with <strong>Decision Trees</strong>, we followed simple clues — things like temperature, weekends, and local events — to predict how much lemonade we might sell. That worked fine, at least for a while.</p><p>But life isn’t that simple, is it?<br> What if it suddenly rains? Or there’s a school holiday nearby? What about the days when you ran a discount or handed out free samples?</p><p>Relying on just one <strong>Decision Tree</strong> to capture all these changing clues can be risky. One misleading pattern can throw the entire prediction off. That’s where the <strong>Random Forest</strong> steps in — it builds a team of many trees, each analyzing the problem from a slightly different angle. Together, they create a prediction that’s far more balanced and trustworthy.</p><h3>🌳 How Does a Random Forest Work?</h3><p>Think of a <strong>Random Forest</strong> as a <em>committee of detectives</em>. Each one investigates the same mystery — your lemonade sales — but looks at different combinations of clues.</p><p>One tree might focus more on weather, another on discounts, and another on holidays. Individually, they might make small mistakes. But when their opinions are combined — averaged together — those random errors cancel out.</p><p>That’s the beauty of a Random Forest:</p><blockquote><em>It doesn’t depend on a single voice. It listens to the wisdom of the crowd.</em></blockquote><p>By blending insights from many trees, the forest makes predictions that are stable, reliable, and often surprisingly accurate — especially in noisy, unpredictable real-world data.</p><h3>🔍 Expanding the Dataset: More Clues, More Accuracy</h3><p>As our lemonade stand story continues, our detective gets a little wiser. Last time, we only looked at simple clues — things like temperature, weekends, and events.</p><p>But now, we’re digging deeper. What if we also keep track of <strong>holidays</strong>, <strong>advertising spend</strong>, <strong>discounts</strong>, and even <strong>rainfall</strong>? Each of these clues affects lemonade sales in its own unique way.</p><p>The more clues we gather, the clearer the story becomes — but it also gets more complicated. Some clues might overlap, others might mislead. That’s where the <strong>Random Forest</strong> truly shines — it can explore all these hints together, spotting patterns that a single decision tree might easily miss.</p><p>Let’s create a synthetic dataset to understand the concept more clearly.</p><pre>import numpy as np<br>import pandas as pd<br># Set seed for reproducibility<br>np.random.seed(42)<br>n_points = 41<br># Generate the features<br>temperature = np.random.uniform(25, 45, n_points)  # Random temperatures between 25°C and 45°C<br>weekend = np.random.choice([0, 1], size=n_points)  # Randomly assign whether it&#39;s a weekend (1) or weekday (0)<br>event = np.random.choice([0, 1], size=n_points, p=[0.7, 0.3])  # Randomly assign events with 30% probability<br>holiday = np.random.choice([0, 1], size=n_points, p=[0.8, 0.2])  # Holiday with 20% probability<br>ad_spend = np.random.uniform(100, 1000, n_points)  # Advertisement spend between $100 and $1000<br>discount = np.random.uniform(0, 30, n_points)  # Discount percentages between 0% and 30%<br>rain = np.random.choice([0, 1], size=n_points, p=[0.6, 0.4])  # 40% chance of rain<br><br># Calculate sales based on the new conditions:<br>base_sales = 30 + 2 * temperature  # Base sales formula<br>sales = base_sales + 20 * weekend * (temperature &gt; 35) - 10 * (weekend == 0) * (temperature &lt; 35)<br>sales += 50 * event + 30 * holiday + 0.05 * ad_spend - 2 * discount - 15 * rain<br><br># Put data into a DataFrame<br>df = pd.DataFrame({<br>    &#39;Temperature&#39;: temperature,<br>    &#39;Weekend&#39;: weekend,<br>    &#39;Event&#39;: event,<br>    &#39;Holiday&#39;: holiday,<br>    &#39;Ad_Spend&#39;: ad_spend,<br>    &#39;Discount&#39;: discount,<br>    &#39;Rain&#39;: rain,<br>    &#39;Sales&#39;: sales<br>})<br><br>df.sample(10)</pre><p>📊 <em>Here’s a quick glimpse of our updated dataset:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSUsv2DTIkosPBXw4Z7NgQ.png" /><figcaption>First few rows of dataset</figcaption></figure><p>🌈 <em>To visualize how these clues vary across days:</em></p><pre>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br># Create subplots, 3 rows and 2 columns<br>fig, axes = plt.subplots(4, 2, figsize=(5, 8))<br><br># Plot 1: Sales vs Temperature<br># sns.stripplot(x=&#39;Temperature&#39;, y=&#39;Sales&#39;, data=df,  ax=axes[0, 1], color=&#39;r&#39;)<br>axes[0, 0].scatter(df[&#39;Temperature&#39;], df[&#39;Sales&#39;], color=&#39;b&#39;, s = 3)<br>axes[0, 0].set_title(&#39;Sales vs Temperature&#39;)<br>axes[0, 0].set_xlabel(&#39;Temperature (°C)&#39;)<br>axes[0, 0].set_ylabel(&#39;Sales&#39;)<br><br># Plot 2: Sales vs Weekend<br>sns.stripplot(x=&#39;Weekend&#39;, y=&#39;Sales&#39;, data=df, ax=axes[0, 1], color=&#39;r&#39;, s = 3)<br>axes[0, 1].set_title(&#39;Sales vs Weekend&#39;)<br>axes[0, 1].set_xlabel(&#39;Weekend (1=Yes, 0=No)&#39;)<br>axes[0, 1].set_ylabel(&#39;Sales&#39;)<br><br># Plot 3: Sales vs Event<br>sns.stripplot(x=&#39;Event&#39;, y=&#39;Sales&#39;, data=df, ax=axes[1, 0], color=&#39;g&#39;, s = 3)<br># axes[1, 0].scatter(df[&#39;Event&#39;], df[&#39;Sales&#39;], color=&#39;g&#39;)<br>axes[1, 0].set_title(&#39;Sales vs Event&#39;)<br>axes[1, 0].set_xlabel(&#39;Event (1=Yes, 0=No)&#39;)<br>axes[1, 0].set_ylabel(&#39;Sales&#39;)<br><br># Plot 4: Sales vs Holiday<br>sns.stripplot(x=&#39;Holiday&#39;, y=&#39;Sales&#39;, data=df, ax=axes[1, 1], color=&#39;orange&#39;, s = 3)<br># axes[1, 1].scatter(df[&#39;Holiday&#39;], df[&#39;Sales&#39;], color=&#39;orange&#39;)<br>axes[1, 1].set_title(&#39;Sales vs Holiday&#39;)<br>axes[1, 1].set_xlabel(&#39;Holiday (1=Yes, 0=No)&#39;)<br>axes[1, 1].set_ylabel(&#39;Sales&#39;)<br><br># Plot 5: Sales vs Ad Spend<br>axes[2, 0].scatter(df[&#39;Ad_Spend&#39;], df[&#39;Sales&#39;], color=&#39;purple&#39;, s = 3)<br>axes[2, 0].set_title(&#39;Sales vs Advertisement Spend&#39;)<br>axes[2, 0].set_xlabel(&#39;Advertisement Spend ($)&#39;)<br>axes[2, 0].set_ylabel(&#39;Sales&#39;)<br><br># Plot 6: Sales vs Discount<br>axes[2, 1].scatter(df[&#39;Discount&#39;], df[&#39;Sales&#39;], color=&#39;brown&#39;, s = 3)<br>axes[2, 1].set_title(&#39;Sales vs Discount&#39;)<br>axes[2, 1].set_xlabel(&#39;Discount (%)&#39;)<br>axes[2, 1].set_ylabel(&#39;Sales&#39;)<br><br># Plot 7: Sales vs Rain<br>sns.stripplot(x=&#39;Rain&#39;, y=&#39;Sales&#39;, data=df, ax=axes[3, 0], color=&#39;lightblue&#39;, s = 3)<br># axes[1, 0].scatter(df[&#39;Event&#39;], df[&#39;Sales&#39;], color=&#39;g&#39;)<br>axes[3, 0].set_title(&#39;Sales vs Rain&#39;)<br>axes[3, 0].set_xlabel(&#39;Rain (1=Yes, 0=No)&#39;)<br>axes[3, 0].set_ylabel(&#39;Sales&#39;)<br><br><br># Adjust layout<br>plt.tight_layout()<br><br># Show the plots<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zU3jQiT6gGhn6TZuR2hI8w.png" /><figcaption>Exploratory Visual Analysis</figcaption></figure><h3>🌱 How Trees Are Built in a Random Forest</h3><p>Imagine each tree in the forest as a <strong>detective working on its own case file</strong>. Each one doesn’t have access to all the clues, and each one looks at a slightly different version of the story. This randomness is what makes the forest wise.</p><h4>1️⃣ Bootstrap Sampling: Picking the Clues</h4><p>Before a tree starts investigating, it gets its own set of clues — a random subset of the data. Typically, about <strong>63% of the original dataset</strong> is selected <strong>with replacement</strong>, which means some clues appear multiple times, while others don’t appear at all.</p><p>The clues left out — roughly <strong>37% of the data</strong> — are called <strong>out-of-bag (OOB) data</strong>. These act like a secret test: after the tree makes predictions, we can check how well it did on clues it hasn’t seen yet.</p><p>Let’s see this in action 👇</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.metrics import mean_squared_error<br>from sklearn.utils import resample<br>from sklearn.tree import plot_tree<br><br># 🎯 Assume df is already defined with columns like:<br># [&#39;Temperature&#39;, &#39;Weekend&#39;, &#39;Event&#39;, &#39;Holiday&#39;, &#39;Ad_Spend&#39;, &#39;Discount&#39;, &#39;Rain&#39;, &#39;Sales&#39;]<br><br># Define the features (X) and target (y)<br>X_train = df[[&#39;Temperature&#39;, &#39;Weekend&#39;, &#39;Event&#39;, &#39;Holiday&#39;, &#39;Ad_Spend&#39;, &#39;Discount&#39;, &#39;Rain&#39;]]<br>y_train = df[&#39;Sales&#39;]<br><br># 🌳 Train a small Random Forest<br>clf_rf = RandomForestRegressor(<br>    n_estimators=5,       # 5 trees (detectives)<br>    max_depth=3,          # limit depth so visualization is clear<br>    max_features=3,       # each split considers 3 random features<br>    min_samples_split=7,  # stop splitting when nodes are too small<br>    random_state=43<br>)<br>clf_rf.fit(X_train, y_train)<br><br># Evaluate training performance (for demo only)<br>print(&quot;MSE on training data:&quot;, mean_squared_error(y_train, clf_rf.predict(X_train)))</pre><p>Each tree here has seen only about 63% of the rows (its bootstrap sample).<br>Let’s peek into one of them to visualize this process.</p><pre># 🧩 Inspect how the first tree was trained<br>tree_1 = clf_rf.estimators_[1]<br><br># Get bootstrap indices for this tree<br>bootstrap_indices = resample(<br>    np.arange(X_train.shape[0]),<br>    replace=True,<br>    random_state=clf_rf.estimators_[1].random_state<br>)<br>unique_rows = np.unique(bootstrap_indices)<br><br>print(f&quot;Number of unique rows for tree_1: {len(unique_rows)}&quot;)<br><br># Visualize which data points were used (and how often)<br>fig, axes = plt.subplots(1, 2, figsize=(15, 6))<br>sns.stripplot(x=&#39;Event&#39;, y=&#39;Sales&#39;, data=df.loc[unique_rows], color=&#39;orange&#39;, ax=axes[0])<br>axes[0].set_title(&#39;Unique ~63% Bootstrap Sample (tree_1)&#39;)<br><br>sns.stripplot(x=&#39;Event&#39;, y=&#39;Sales&#39;, data=df.loc[bootstrap_indices], color=&#39;orange&#39;, ax=axes[1])<br>axes[1].set_title(&#39;Final Bootstrap Sample with Repetition (tree_1)&#39;)<br>plt.show()<br><br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAtUrE1OFWhNbWWER7_b6Q.png" /></figure><pre># Compare bootstrap samples across all 5 trees<br>for i in range(5):<br>    bootstrap_indices = resample(<br>        np.arange(X_train.shape[0]),<br>        replace=True,<br>        random_state=clf_rf.estimators_[i].random_state<br>    )<br>    sns.stripplot(x=&#39;Event&#39;, y=&#39;Sales&#39;, data=df.loc[bootstrap_indices], color=&#39;orange&#39;)<br>    plt.title(f&#39;Bootstrap Sample (Detective {i+1})&#39;)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*L1Othwg7Cng2m97D113fYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*Xp3kVJTRNBq2iHU9UiMy2w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*SfCWO_IG-JmZMlIDjhHh1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*FuzkUBM2NNhTIwA6mYOi9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*4ET3Erspp9BvyWpsRnyBGA.png" /></figure><p>Each “detective” has seen a different combination of evidence — the first layer of randomness in our forest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VgrfcWUKxlVfV0MAiM6aMQ.png" /><figcaption>Prototype example: Bootstrapped sample data with repeated data points</figcaption></figure><h4>2️⃣ Random Feature Selection: Focusing the Investigation</h4><p>Unlike a single Decision Tree, which looks at <strong>all the clues</strong> when making a decision, a Random Forest tree considers only a <strong>random subset of features</strong> at each split.</p><ul><li>For <strong>classification</strong>, typically √M features are chosen.</li><li>For <strong>regression</strong>, roughly M/3 features are selected (you can tweak this).</li></ul><p>By forcing each detective to focus on different clues, we ensure that no two trees are exactly alike — and diversity among the trees is what gives the forest its strength.</p><p>(In our code above, max_features=3 controlled this randomness.)</p><h4>3️⃣ Growing the Trees: When to Stop</h4><p>Each tree grows node by node, following rules to find the “best split”:</p><ul><li>Using <strong>Mean Squared Error (MSE)</strong> for regression or <strong>Gini Impurity</strong> for classification.</li></ul><p>Stopping when:</p><ul><li>A node has too few samples</li><li>A split doesn’t improve the prediction</li><li>A maximum depth is reached</li></ul><p>Think of it as the detective knowing when to stop chasing dead-end clues.</p><p>Let’s visualize what one of our detectives discovered:</p><pre># 🌳 Visualize one of the trained trees<br>plt.figure(figsize=(20, 10))<br>plot_tree(<br>    clf_rf.estimators_[4],<br>    feature_names=X_train.columns,<br>    filled=True,<br>    rounded=True,<br>    fontsize=12<br>)<br>plt.title(&quot;Visualization of Tree 3 (Detective #3)&quot;)<br>plt.show()</pre><p>Each node shows which feature the tree used to split the data, the threshold, and the resulting predictions.</p><h4>4️⃣ Randomness: The Secret Sauce</h4><p>The combination of <strong>random row sampling</strong> and <strong>random feature selection</strong> ensures that each tree is unique.</p><p>Why does this matter? Because it reduces overfitting. A single Decision Tree might memorize the training data too well, but the forest, by listening to many slightly different detectives, learns the <strong>general story</strong>.</p><p>* What would the individual trees look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*76UwhrhE6-Q7f-5bD9YRqQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*9xzHAW3G3qIHwTmvXcTmWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*pkO1Uo63p7vCnx4DNCLqyA.png" /></figure><h4>🎯 How Do Predictions Work?</h4><p>Once all the trees have finished investigating, it’s time for the forest to <strong>reach a verdict</strong>.</p><p>Imagine it like this: each detective has examined the clues in their own way and comes up with a prediction for lemonade sales on a new day. Some detectives are optimistic, others are cautious, and a few might overestimate the crowd at your stand.</p><p>For instance:</p><ul><li>Tree 1 predicts <strong>73.88 cups</strong></li><li>Tree 2 predicts <strong>78.01 cups</strong></li><li>Tree 3 predicts <strong>93.56 cups</strong><br> …and so on.</li></ul><p>The Random Forest doesn’t just pick one opinion — it <strong>listens to all the detectives</strong>. By averaging their predictions, the forest produces a <strong>final, balanced, and more reliable estimate</strong>.</p><p>Think of it as consulting a council: one tree might be wrong, but together, the wisdom of the group leads to a far more accurate decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8uCI57dKLvZo5RbkZaPC3w.png" /><figcaption>Prediction formula for Regression</figcaption></figure><p>Let’s bring this to life with Python 👇</p><pre># 🎯 Compare predictions from individual trees vs. the full forest<br><br># Each tree (detective) makes its own prediction<br>df[&#39;prediction_tree0&#39;] = clf_rf.estimators_[0].predict(X_train)<br>df[&#39;prediction_tree1&#39;] = clf_rf.estimators_[1].predict(X_train)<br>df[&#39;prediction_tree2&#39;] = clf_rf.estimators_[2].predict(X_train)<br>df[&#39;prediction_tree3&#39;] = clf_rf.estimators_[3].predict(X_train)<br>df[&#39;prediction_tree4&#39;] = clf_rf.estimators_[4].predict(X_train)<br><br># The Random Forest&#39;s overall prediction (the &quot;council verdict&quot;)<br>df[&#39;prediction&#39;] = clf_rf.predict(X_train)<br><br># Compute the simple average manually (should match forest prediction)<br>df[&#39;prediction_avg&#39;] = df[<br>    [&#39;prediction_tree0&#39;, &#39;prediction_tree1&#39;, &#39;prediction_tree2&#39;, &#39;prediction_tree3&#39;, &#39;prediction_tree4&#39;]<br>].mean(axis=1)<br><br># 🎨 Visualize a few sample predictions<br>df_sample = df.sample(5, random_state=42)  # pick 5 random days<br><br>plt.figure(figsize=(10, 6))<br>plt.scatter(df_sample[&#39;Temperature&#39;], df_sample[&#39;Sales&#39;], color=&#39;b&#39;, label=&#39;Actual Sales&#39;)<br><br># Show each tree’s prediction (light, transparent points)<br>plt.scatter(df_sample[&#39;Temperature&#39;], df_sample[&#39;prediction_tree0&#39;], marker=&#39;x&#39;, label=&#39;Tree 1&#39;, alpha=0.4)<br>plt.scatter(df_sample[&#39;Temperature&#39;], df_sample[&#39;prediction_tree1&#39;], marker=&#39;x&#39;, label=&#39;Tree 2&#39;, alpha=0.4)<br>plt.scatter(df_sample[&#39;Temperature&#39;], df_sample[&#39;prediction_tree2&#39;], marker=&#39;x&#39;, label=&#39;Tree 3&#39;, alpha=0.4)<br>plt.scatter(df_sample[&#39;Temperature&#39;], df_sample[&#39;prediction_tree3&#39;], marker=&#39;x&#39;, label=&#39;Tree 4&#39;, alpha=0.4)<br>plt.scatter(df_sample[&#39;Temperature&#39;], df_sample[&#39;prediction_tree4&#39;], marker=&#39;x&#39;, label=&#39;Tree 5&#39;, alpha=0.4)<br><br># The forest&#39;s final, averaged prediction (bold squares)<br>plt.scatter(df_sample[&#39;Temperature&#39;], df_sample[&#39;prediction_avg&#39;], <br>            marker=&#39;s&#39;, color=&#39;black&#39;, label=&#39;Average Prediction&#39;, s=100)<br><br>plt.xlabel(&#39;Temperature (°C)&#39;)<br>plt.ylabel(&#39;Sales (cups)&#39;)<br>plt.title(&#39;Actual vs Predicted Sales — Individual Trees vs Forest&#39;)<br>plt.legend(loc=&#39;upper left&#39;, bbox_to_anchor=(1, 1))<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*XRR1GxvPc3W5Lbzhulocsg.png" /></figure><h3>🌳 Why This Works</h3><p>A single tree can easily be <strong>biased</strong> or <strong>noisy</strong>, especially if it happens to get an unrepresentative bootstrap sample.<br> But when you aggregate many trees, those random errors tend to cancel each other out.</p><p>That’s the power of <strong>ensemble learning</strong>:</p><blockquote><em>“Alone, they may be wrong — together, they are almost always right.”</em></blockquote><h4>⚠️ Pitfalls and Misconceptions</h4><p>Even the wisest forests have their challenges. While the <strong>Random Forest</strong> is a powerful predictor, it isn’t perfect:</p><ol><li><strong>Overfitting</strong><br>Although less prone than a single tree, the forest can still be misled if parameters like <strong>max depth</strong> aren’t set thoughtfully. Even a group of detectives can get carried away if they overanalyse the clues.</li><li><strong>Interpretability:</strong><br>With many trees working together, it can be hard to trace exactly why a certain prediction was made. It’s like asking each detective to explain their thought process in detail — the answer can get complicated fast.</li><li><strong>Computation:</strong><br>Building and combining predictions from dozens or hundreds of trees requires more computing power. The more clues and detectives you have, the heavier the forest becomes.</li></ol><h3>🌳 Conclusion: The Wisdom of the Forest</h3><p>At its heart, the <strong>Random Forest Regressor</strong> is about collaboration. Each tree looks at different clues, learns patterns independently, and then the forest combines their insights. The result? More accurate predictions and hidden patterns that simpler models might miss.</p><p>But, like any team, complexity comes with trade-offs. The forest is harder to interpret, and it demands more computational resources. The key is to balance <strong>power and simplicity</strong>, knowing when to rely on a single tree and when to call in the entire forest.</p><p>In the end, Random Forest teaches us a lesson beyond machine learning: sometimes, <strong>collective wisdom beats a single voice</strong>, even in the unpredictable world of lemonade sales.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e835fffa6a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Tree Regressor: The Unsung Foundation for Mastering Machine Learning using Elementary…]]></title>
            <link>https://medium.com/@algostalk/decision-tree-regressor-the-unsung-foundation-for-mastering-machine-learning-using-elementary-a036911f64cf?source=rss-06c43f99303c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a036911f64cf</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression-modeling]]></category>
            <category><![CDATA[tree-based-model]]></category>
            <category><![CDATA[decision-tree]]></category>
            <dc:creator><![CDATA[AlgoStalk]]></dc:creator>
            <pubDate>Sun, 01 Dec 2024 03:09:04 GMT</pubDate>
            <atom:updated>2024-12-01T03:09:04.890Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Decision Tree Regressor: The Unsung Foundation for Mastering Machine Learning using Elementary Maths</strong></h3><p>If you prefer visual understanding, check out this our link on the <a href="https://youtu.be/SQbXSpOMewc">Decision Tree Regressor!</a></p><p>Imagine you have a pile of data points and a list of features. You must decide how to organise the data points to make solving the case as easy as possible. This is exactly how a decision tree works — it helps us break down complex problems into smaller, simpler steps by deciding where to split the data.</p><p>In this article, we’ll explore:</p><ol><li>Why do We Need Decision Trees?</li><li>What is a Decision Tree</li><li>How decision trees decide where to split and optimise splits using math</li><li>The meaning of their parameters.</li><li>Why decision trees are foundational but often underused.</li><li>Their pros and cons.</li><li>Where Decision Tree Shines</li><li>Common pitfalls and misconceptions.</li></ol><p>Grab your detective hat, and let’s dive in! 🕵️♂️</p><p><strong>Create a Dummy Data</strong></p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from mpl_toolkits.mplot3d import Axes3D<br>import pandas as pd<br><br># Generate sample data with 20 observations<br>np.random.seed(40)<br>n_points = 10<br>temperature = np.random.uniform(25, 45, n_points)  # Random temperatures between 25°C and 45°C<br>weekend = np.random.choice([0, 1], size=n_points)  # Randomly assign whether it&#39;s a weekend (1) or weekday (0)<br>event = np.random.choice([0, 1], size=n_points, p=[0.7, 0.3])  # Randomly assign events with 30% probability of occurring<br><br># Calculate sales based on conditions:<br>base_sales = 30 + 2 * temperature  # Base sales formula<br>sales = base_sales + 20 * weekend * (temperature &gt; 35) - 10 * (weekend == 0) * (temperature &lt; 35)<br>sales += 50 * event  # Add jump in sales if there’s an event<br><br># Create a DataFrame<br>data = {<br>    &#39;Temperature (°C)&#39;: temperature,<br>    &#39;Weekend (1=Yes, 0=No)&#39;: weekend,<br>    &#39;Event (1=Yes, 0=No)&#39;: event,<br>    &#39;Sales&#39;: sales<br>}<br><br>df = pd.DataFrame(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Es4ndV29lL-i70cLa4S8A.png" /></figure><p><strong>Visualise your Data</strong></p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br><br># Create subplots<br>fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))<br><br># Plot 1: Sales vs Temperature (Categorized by Weekend/Weekday)<br>ax1.scatter(df[&#39;Temperature (°C)&#39;][df[&#39;Weekend (1=Yes, 0=No)&#39;] == 1],<br>            df[&#39;Sales&#39;][df[&#39;Weekend (1=Yes, 0=No)&#39;] == 1],<br>            color=&#39;red&#39;, label=&#39;Weekend&#39;, marker=&#39;o&#39;)<br>ax1.scatter(df[&#39;Temperature (°C)&#39;][df[&#39;Weekend (1=Yes, 0=No)&#39;] == 0],<br>            df[&#39;Sales&#39;][df[&#39;Weekend (1=Yes, 0=No)&#39;] == 0],<br>            color=&#39;blue&#39;, label=&#39;Weekday&#39;, marker=&#39;x&#39;)<br>ax1.set_xlabel(&#39;Temperature (°C)&#39;)<br>ax1.set_ylabel(&#39;Lemonades Sold&#39;)<br>ax1.set_title(&#39;Sales vs Temperature (Weekend vs Weekday)&#39;)<br>ax1.set_ylim(65, 180)<br>ax1.legend()<br><br># Plot 2: Sales vs Temperature (Categorized by Event Presence)<br>ax2.scatter(df[&#39;Temperature (°C)&#39;][df[&#39;Event (1=Yes, 0=No)&#39;] == 1],<br>            df[&#39;Sales&#39;][df[&#39;Event (1=Yes, 0=No)&#39;] == 1],<br>            color=&#39;green&#39;, label=&#39;With Event&#39;, marker=&#39;o&#39;)<br>ax2.scatter(df[&#39;Temperature (°C)&#39;][df[&#39;Event (1=Yes, 0=No)&#39;] == 0],<br>            df[&#39;Sales&#39;][df[&#39;Event (1=Yes, 0=No)&#39;] == 0],<br>            color=&#39;purple&#39;, label=&#39;No Event&#39;, marker=&#39;x&#39;)<br>ax2.set_xlabel(&#39;Temperature (°C)&#39;)<br>ax2.set_ylabel(&#39;Lemonades Sold&#39;)<br>ax2.set_title(&#39;Sales vs Temperature (Event Presence)&#39;)<br>ax2.legend()<br><br># Adjust layout for better spacing<br>plt.tight_layout()<br><br># Show the plots<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nx4wepe7DbzqqJvxvT_D4A.png" /></figure><h3>1. Why Do We Need Decision Tree Regressors?</h3><p>So, why do we need Decision Tree Regressors? Imagine you have a traditional detective — a detective who can only follow a straight path. They do great when all the clues line up neatly, leading directly to a single conclusion. This is like using a traditional model, such as Linear Regression, which assumes that all clues have a simple, straight-line relationship to the outcome.</p><p>But what if the case is more complicated? What if the clues twist and turn, and different suspects become more suspicious at different times? That’s where our clever detective, the Decision Tree Regressor, comes into play. This detective doesn’t rely on a single path; instead, they carefully examine each clue’s behavior, solving mysteries involving complex, non-linear relationships.</p><p>Take, for example, investigating sales at a lemonade stand. Our suspects are the temperature, the day of the week, and special events. But the link between these suspects and the sales isn’t straightforward:</p><ul><li>Sales soar on weekends when it’s hot and there’s a local event. But, they drop if it’s cool — even on weekends.</li><li>On weekends with moderate temperatures, sales still climb if there’s an event but remain steady without one.</li><li>On weekdays, sales are usually lower, but they can spike if an event happens, even when it’s hot outside.</li></ul><h3>2. What Is a Decision Tree Regressor?</h3><p>So, what exactly is a Decision Tree Regressor? Picture it as the detective’s decision-making flowchart. Each decision point — or ‘node’ — is like an interrogation with a suspect, a key feature in our investigation. As we move down the branches, we’re gathering more information, narrowing down the possibilities one clue at a time.</p><p>Eventually, we reach an end point, called a ‘leaf,’ where our detective has gathered enough clues to make a prediction. It’s like concluding the case: after considering all the evidence, a decision is reached. The Decision Tree is built by picking out the most suspicious suspects first, examining each one in detail, and gradually focusing on the most relevant information until a prediction is confidently made.</p><h3>3. How It Works: Step-by-Step</h3><h3>3.1 Start Simple: One Dimension</h3><p>Let’s start simple, focusing on just one suspect in our investigation — temperature. We want to see how this suspect alone might affect lemonade sales. Imagine our detective is only asking one question: ‘What is the temperature?’</p><p>To do this, we look at all the temperatures in our data, organising them in order from the coolest to the hottest days.</p><pre>df_sorted = df.sort_values(by = [&#39;Temperature (°C)&#39;]).copy()<br>temperature = df_sorted[&#39;Temperature (°C)&#39;].values<br>sales = df_sorted[&#39;Sales&#39;].values<br><br>plt.scatter(temperature, sales, c=&#39;#0033ff&#39;, s = 100)<br># plt.plot(temperature, sales, c = &#39;purple&#39;, linestyle = &#39;--&#39;, alpha = 0.2)<br>plt.ylabel(&#39;Lemonades Sold&#39;)<br>plt.xlabel(&#39;Temperature (°C)&#39;)<br>plt.title(&#39;Impact of Temperature on Lemonade Sales&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*wu4Wh3tIYzrlHvdiwmvAMg.png" /></figure><p>Then, we search for the best places to split our investigation. These splits are like decision points where the temperature changes just enough to give us a different clue about sales.</p><pre>def get_split_points(df, feature = &#39;Temperature (°C)&#39;):<br>  X = df[feature].values.reshape(-1, 1)  # Feature: temperature, reshaped for modeling<br>  y = df[&#39;Sales&#39;].values  # Target: sales<br><br>  # Sort temperature for consistent split points<br>  sorted_temp = np.sort(X.flatten())  # Flattening to get a 1D array for sorting<br><br>  # Calculate potential split points as midpoints between sorted temperatures<br>  split_points = [(sorted_temp[i] + sorted_temp[i + 1]) / 2 for i in range(len(sorted_temp) - 1)]<br>  return(split_points)<br>split_points = get_split_points(df)<br>split_points</pre><pre>plt.scatter(temperature, sales, c=&#39;#0033ff&#39;, s = 100)<br># plt.plot(temperature, sales, c = &#39;purple&#39;, linestyle = &#39;--&#39;, alpha = 0.2)<br>for v in split_points:<br>  plt.axvline(x=v, color=&#39;red&#39;, linestyle=&#39;--&#39;, alpha = 0.5)<br>plt.ylabel(&#39;Lemonades Sold&#39;)<br>plt.xlabel(&#39;Temperature (°C)&#39;)<br>plt.title(&#39;Potential Splits of Temperature on Lemonade Sales&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*naNSPWlcBPhjpRL9oVF-qg.png" /></figure><p>By finding the ideal split points between different temperature values, we can see how temperature alone influences the outcome — whether it’s a cool, breezy day or a scorching hot afternoon. Each split brings us one step closer to understanding the bigger picture of our lemonade sales.</p><h4>3.1.1. Determining the Split Point: Minimizing Cost</h4><p>Now, our detective needs to decide the best split point. How do they make that decision? They look at every potential split and calculate something called the Weighted Mean Squared Error — or Weighted MSE. Here’s how it works:</p><p>Each potential split divides the clues — the data — into two groups: a left group and a right group. For each group, the detective calculates the Mean Squared Error, or MSE. This MSE measures how well the detective’s prediction matches the clues in each group. To make the prediction, the detective uses the average value of each group. In other words, they assume that every clue in the group can be summarized by the group’s average.</p><p>But the detective doesn’t stop there. They need to consider how many clues are in each group. So, they calculate the Weighted MSE, which takes into account the number of clues in the left and right groups. Here’s the formula they use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-6Eu0PshxGlYtmNrVUNLw.png" /></figure><h4>3.1.2. Deciding the First Sleep</h4><p>To truly understand our detective’s method, let’s follow them as they solve for a single split. Imagine they’ve picked a potential split point, dividing the temperature data into two groups: one cooler and one warmer. Now, the detective wants to see if this split is meaningful — how well does it help predict lemonade sales?</p><p>First, they calculate the prediction error for each group. To do this, they use the group’s average as the prediction, like a detective assuming all clues in that group point to the same result. The detective calculates how far off each clue is from the group’s prediction, squaring the difference and to avoid any negative numbers.</p><p>Then, they calculate the average error in each group — the Mean Squared Error (MSE). These MSE values tell the detective how accurate their predictions are for the cooler group and the warmer group.</p><p>But not all groups are equal in size! To get a fair evaluation, the detective considers the number of clues in each group. They calculate a Weighted MSE by balancing the errors based on how many clues fell into the cooler group versus the warmer group.</p><h4><strong>Assumption</strong></h4><p>Assume the threshold to analyse the dataset is Temperature = 33.58</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZP2F_eauqcUhpMPuq9y9Tg.png" /></figure><p>Thus, the mathematical calculations to get Weighted MSE would be as follows:</p><pre># For Left group:<br>Predictions = (146.30 + 82.21 + 131.49 + 82.15) / 4 <br>               #  = 110.54 (approximately); Mean of the data points falling under this group<br>MSE_left = [(146.3 - 110.54)**2 + (82.21 - 110.54)**2 + (131.49 - 110.54)**2 + (82.15 - 110.54)**2] / 4 # = 831.56 (approximately)<br>weight_left = 0.4 # (number of data points = 4 and total number of data points = 10)</pre><pre>Similarly, MSE_right = 691.54 and <br>weight_right = 0.6</pre><pre>Weighted_MSE = (0.4 * 831.56) + (0.6 * 691.54) = 747.548</pre><h4>3.1.3 Analysing all the splits</h4><p>In similar fashion, detective analyses the Weighted MSE for all the potential splits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Hye6Yinszp6AGnRxPzbag.png" /></figure><p>By balancing the errors of both groups, the detective finds the split that has the lowest Weighted MSE (Temperature = 38.1). This means they’ve found the best possible way to split the data, minimising prediction errors while accounting for all the clues they’ve gathered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x7f3yWU7f3oBOEqIlavPnA.png" /></figure><p>Now that we’ve found our first split, our detective is ready to show us the first step in their decision-making process. Imagine our data as a tree — split right down the middle based on the chosen temperature point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JL2Y374tdBqqbBMoxU6rbg.png" /></figure><p>In the left branch, we have 7 samples out of 10, with an average value of 108.028. This means that if a day falls on this side of the split, our detective predicts a sale around 108.028. In the right branch, there are 3 remaining samples, averaging to 150.021. So, on days that land here, the prediction would be closer to 150.021, based on the temperature.</p><p>But the investigation doesn’t stop here! Now, the detective zooms in on the left group — those 7 samples. To the detective, this left group is like a fresh case, a new dataset to solve. They’ll apply the same method, searching for the best split just as they did before. The only difference is that now, these 7 samples become the detective’s entire dataset at the current depth.</p><p>Using exact same methodology, best split is determined for this node:</p><pre>Left Group:  [82.2146416]<br>Prediction:  82.2146416044744<br>MSE_left:  0.0<br>Weight_left:  0.14285714285714285<br>Right Group:  [146.30748112 131.49220739  88.01402348  82.15649221 121.05598097<br> 104.95248852]<br>Prediction:  112.32977894906334<br>MSE_right:  525.6561868748252<br>Weight_right:  0.8571428571428571<br>Weighted MSE:  450.5624458927073</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFqfj2JKILy0ki_h1Ivzig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*71oI_y7Qxz_TzoK6ZEn5wQ.png" /></figure><p>The Detective will keep on splitting the data set until there’s no room left to split the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mT0p5Uzp8VQ-CFmEHPOYIg.png" /></figure><h3>4. Stopping the Interrogation: Depth and Minimum Samples</h3><p>By this stage, model has learned everything or i should say more than required, if trees are let to grow, they result in overfitting. To avoid overcomplicating the investigation:</p><p>The detective can set a maximum depth (how many times to split). They can also set a minimum number of clues (data points) required to make a split. This helps ensure that the tree doesn’t overfit, making a story that fits the training clues perfectly but struggles with new clues.</p><h3>5. Why Decision Trees Are Building Blocks</h3><p>Decision Trees are rarely used alone because:</p><ul><li>They overfit: They memorise the training data too well.</li><li>They’re unstable: Small changes in data can create a completely different tree.</li></ul><p>However, they are critical building blocks for ensemble models like Random Forest and XGBoost, which combine multiple trees to create stronger models.</p><p>Understanding Decision Trees helps you grasp these advanced techniques.</p><h3>6. Pros and Cons of Decision Trees</h3><h4>Pros:</h4><ul><li>Easy to understand and visualise.</li><li>Handles both numerical and categorical data.</li><li>No need for feature scaling.</li></ul><h4>Cons:</h4><ul><li>Overfits on noisy data.</li><li>Sensitive to small changes in data.</li></ul><h3>7. Where Decision Trees Shine</h3><ul><li>Interpretable Models: Easy to explain decisions.</li><li>Exploratory Analysis: Quickly identifies important features.</li></ul><h3>8. Common Pitfalls and Misconceptions</h3><ul><li>Bigger Trees Are Always Better</li><li>Trees Always Work Well</li></ul><h3>Final Decision Tree with max_depth = 3 and min_samples_split = 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*BX0UhtTMTQjBIc5zptI-Ag.png" /></figure><h3>Conclusion: Understanding Decision Trees</h3><p>Decision Trees are powerful yet simple models that help us make predictions by splitting data into smaller, more manageable groups. By focusing on minimising errors through splits based on features, the tree learns to predict a target value at the end of each branch or leaf.</p><p>Through this explanation, we’ve seen how:</p><ul><li>Splits are made by evaluating which feature best divides the data to reduce errors.</li><li>The leaf values are calculated as the average target value of the data points within that leaf.</li><li>Decision Trees form the foundation for more complex models, such as Random Forests and XGBoost, and understanding them opens the door to mastering these advanced algorithms.</li></ul><p>While Decision Trees can overfit and are sensitive to changes in the data, they shine in simple, interpretable applications and are crucial to machine learning workflows. By understanding their mechanics, strengths, and limitations, you’ll be better equipped to use them effectively, whether on their own or as part of a more advanced ensemble method.</p><p>As you explore deeper into machine learning, remember that Decision Trees offer a clear, visual approach to understanding how models make predictions — an essential skill for tackling more complex tasks in AI and data science!</p><p>If you prefer visual understanding, check out this link on the <a href="https://youtu.be/SQbXSpOMewc">Decision Tree Regressor!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a036911f64cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Huber Regression: A Smarter Approach to Handle Outliers in Machine Learning]]></title>
            <link>https://medium.com/@algostalk/huber-regression-a-smarter-approach-to-handle-outliers-in-machine-learning-7be42ef07cdd?source=rss-06c43f99303c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7be42ef07cdd</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[outliers]]></category>
            <dc:creator><![CDATA[AlgoStalk]]></dc:creator>
            <pubDate>Wed, 06 Nov 2024 03:31:57 GMT</pubDate>
            <atom:updated>2024-11-06T03:31:57.509Z</atom:updated>
            <content:encoded><![CDATA[<p>If you prefer visual understanding, check out my 8 mins video link on the <a href="https://youtu.be/C6Z8Ps23Omw"><strong>Huber Regressor!</strong></a></p><h3>Introduction 🌟</h3><p>In traditional linear regression, extreme data points — known as outliers — can heavily skew predictions. But what if we could create a model that’s less sensitive to these outliers without entirely ignoring them? Enter <strong>Huber Regression</strong>: a hybrid solution that adapts to outliers while preserving the accuracy of regular data points. This article will guide you through how Huber Regression works, the math behind it, and how to implement it with visualisations.</p><h3>1. What’s the Problem with Outliers in Linear Regression? 🤔</h3><p><strong>Outliers</strong> are data points that differ significantly from the majority of a dataset. While sometimes informative, outliers can disrupt the prediction quality in standard regression models.</p><h4>Example Visualization: Outliers and Linear Regression</h4><p>Here’s a visual comparison between a simple linear regression model and a dataset with outliers. Let’s use Python to show how these outliers can affect the regression line.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.linear_model import LinearRegression<br><br># Generate synthetic data<br>np.random.seed(42)<br>X = np.random.normal(0, 1, 50).reshape(-1, 1)<br>y = 2 * X.flatten() + np.random.normal(0, 0.1, 50)<br><br># Add outliers<br>X_outliers = np.append(X, [[1.5], [2], [2.5]])<br>y_outliers = np.append(y, [10, 12, 14])<br><br># Plot Linear Regression with and without outliers<br>model = LinearRegression()<br>plt.figure(figsize=(10, 6))<br><br># Without outliers<br>model.fit(X, y)<br>plt.scatter(X, y, color=&quot;blue&quot;, label=&quot;Data without Outliers&quot;)<br>plt.plot(X, model.predict(X), color=&quot;black&quot;, label=&quot;Regression Line (No Outliers)&quot;)<br><br># With outliers<br>model.fit(X_outliers.reshape(-1, 1), y_outliers)<br>plt.scatter(X_outliers, y_outliers, color=&quot;red&quot;, label=&quot;Data with Outliers&quot;)<br>plt.plot(X_outliers, model.predict(X_outliers.reshape(-1, 1)), color=&quot;red&quot;, linestyle=&quot;--&quot;, label=&quot;Regression Line (With Outliers)&quot;)<br><br>plt.legend()<br>plt.xlabel(&quot;X&quot;)<br>plt.ylabel(&quot;y&quot;)<br>plt.title(&quot;Effect of Outliers on Linear Regression&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*h8qybYsUp75FHWrWQjMhKQ.png" /></figure><h3>2. Introducing Huber Loss Function 📉</h3><p>Huber Regression modifies the linear regression loss function to reduce the impact of outliers. In its core, <strong>Huber Loss</strong> combines the strengths of Mean Squared Error (MSE) and Mean Absolute Error (MAE), switching from MSE to MAE when errors exceed a specific threshold, known as epsilon (𝜖).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AulU33vPIFI2qYxV_AKYcw.png" /></figure><h4>Visualising Huber Loss vs. MSE and MAE</h4><p>We can compare the loss functions to see how Huber Loss adjusts error handling based on 𝜖.</p><pre>def huber_loss(y_true, y_pred, epsilon=1):<br>    error = y_true - y_pred<br>    loss = np.where(np.abs(error) &lt;= epsilon, 0.5 * error ** 2, epsilon * (np.abs(error) - 0.5 * epsilon))<br>    return loss<br><br># Plotting Huber Loss vs. MSE and MAE<br>epsilon = 1<br>errors = np.linspace(-3, 3, 100)<br>huber_losses = [huber_loss(e, 0, epsilon) for e in errors]<br>mse_losses = 0.5 * errors ** 2<br>mae_losses = np.abs(errors)<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(errors, mse_losses, label=&quot;MSE Loss&quot;, color=&quot;blue&quot;)<br>plt.plot(errors, mae_losses, label=&quot;MAE Loss&quot;, color=&quot;green&quot;)<br>plt.plot(errors, huber_losses, label=f&quot;Huber Loss (epsilon={epsilon})&quot;, color=&quot;red&quot;)<br>plt.xlabel(&quot;Error&quot;)<br>plt.ylabel(&quot;Loss&quot;)<br>plt.legend()<br>plt.title(&quot;Comparison of Loss Functions: MSE, MAE, and Huber Loss&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*ZA6I7UXFI4mWmaeWu_9KQA.png" /></figure><h3>3. How Huber Regression Works Step-by-Step 🔍</h3><p>Huber Regression applies the Huber loss function iteratively, using several parameters to fine-tune the model:</p><ul><li><strong>Epsilon (𝜖)</strong>: the threshold where Huber Loss switches from quadratic to linear.</li><li><strong>Alpha</strong>: acts as a regularisation term, minimising the impact of large coefficients and helping with model generalisation.</li><li><strong>Learning Rate</strong>: controls the step size during coefficient updates, preventing the model from overshooting.</li><li><strong>Median Absolute Deviation (MAD)</strong>: estimates the scale of the data, particularly useful when there are outliers.</li></ul><h4>Example: Step-by-Step Visualisation of the Huber Loss and Model Update Process</h4><p>To understand <strong>Huber Regression</strong>, we first need to discuss two key concepts: <strong>Median Absolute Deviation (MAD)</strong> and the <strong>scaling of residuals</strong>. Huber Regression combines the benefits of <strong>Mean Squared Error (MSE)</strong> and <strong>Mean Absolute Error (MAE)</strong>, allowing for robust regression by minimising the influence of outliers. Let’s break down the process step by step using the provided code.</p><h3>Step 1: Initialize Parameters</h3><p>In Huber Regression, we start by defining parameters that will guide the regression process:</p><pre>epsilon = 1.35  # Threshold for Huber loss<br>alpha = 0.0001  # Regularization strength<br>max_iter = 100  # Maximum iterations<br>tol = 1e-5  # Tolerance for convergence<br>n_samples, n_features = X_train_with_intercept.shape<br><br># Initialize coefficients (including intercept)<br>coef = np.zeros(n_features)</pre><ul><li><strong>epsilon</strong>: This value sets the threshold for determining which residuals are treated as outliers.</li><li><strong>alpha</strong>: This controls the amount of regularization applied to the regression model.</li><li><strong>max_iter</strong> and <strong>tol</strong>: These determine how many times the algorithm will run and when to stop if changes are minimal.</li></ul><h3>Step 2: Generate Synthetic Data</h3><p>We’ll generate a dataset with a linear relationship, introduce some outliers, and prepare it for the Huber Regression.</p><pre># Step 1: Generate synthetic data<br>np.random.seed(42)  # For reproducibility<br>X = np.random.normal(0, 1, 50).reshape(-1, 1)  # 50 samples, 1 feature<br>y = 2 * X.flatten() + np.random.normal(0, 0.1, 50)  # Linear relation with noise<br><br># Add outliers<br>X_outliers = np.append(X, [[1.5], [2], [2.5]])<br>y_outliers = np.append(y, [10, 12, 14])<br><br># Combine the data into a DataFrame for easier plotting<br>df_sample_train_10 = pd.DataFrame({&#39;carat&#39;: X_outliers.flatten(), &#39;price&#39;: y_outliers})<br><br># Prepare the data for Huber Regression<br>X_train_with_intercept = np.hstack((np.ones((X_outliers.shape[0], 1)), X_outliers))  # Add intercept<br>y_train = y_outliers</pre><h3>Step 3: Visualising Data</h3><p>We visualise our training data to see how carat relates to price.</p><pre>sns.scatterplot(data=df_sample_train_10, x=&#39;carat&#39;, y=&#39;price&#39;, color=&#39;#304FFE&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*WrJedgbRW8KfNrpku3R17Q.png" /></figure><h3>Step 4: Compute Predictions and Residuals</h3><p>Next, we compute the predictions using the current coefficients and calculate the residuals:</p><pre>predictions = X_train_with_intercept @ coef  # Predictions with intercept<br>residuals = y_train - predictions</pre><p>Since the initialised coefficients were 0*x + 0, all predictions will be 0 for the first iteration and hence all residuals will equal their respective y values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*PmheelXEATcjrh8zBsLnxA.png" /></figure><h3>Step 5: Visualise Residuals</h3><p>To understand the distribution of residuals, we can plot them against the carat values:</p><pre>df_sample_train_10[&#39;residuals&#39;] = residuals<br>df = pd.DataFrame(df_sample_train_10)<br><br># Sort the DataFrame by carat value<br>df_sorted = df.sort_values(by=&#39;carat&#39;)<br><br># Create a bar graph of the residuals<br>plt.figure(figsize=(10, 6))<br>bars = plt.bar(df_sorted[&#39;carat&#39;].astype(str), df_sorted[&#39;residuals&#39;], color=&#39;#FF3D00&#39;, alpha=0.7)<br><br># Annotate the bars with their values<br>for bar in bars:<br>    yval = bar.get_height()<br>    plt.text(bar.get_x() + bar.get_width()/2, yval, f&#39;{int(yval)}&#39;, ha=&#39;center&#39;, va=&#39;bottom&#39;)<br><br>plt.xlabel(&#39;Carat Value&#39;)<br>plt.ylabel(&#39;Residual Value&#39;)<br>plt.title(&#39;Residuals Bar Graph Sorted by Carat Value&#39;)<br>plt.xticks(rotation=45)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*co1nB358vLDGVVKz7zazig.png" /></figure><h3>Step 6: Calculate Median Absolute Deviation (MAD)</h3><p>Now, we compute the MAD, which helps us assess the spread of residuals.</p><p>Median Absolute Deviation (MAD): This method provides a robust measure of variability in the data. The MAD is defined as the median of the absolute deviations from the median of the dataset. This approach is particularly useful for robust statistics because it is less sensitive to outliers than standard deviation.</p><pre>MAD = np.median(np.abs(residuals))</pre><p>Median Absolute Deviation (MAD): 1.227684876170789</p><h3>Step 7: Scale Residuals</h3><p>Scaling the residuals helps to make them more uniform:</p><pre>scale = MAD / 0.6745  # Scale<br>scaled_residuals = residuals / scale</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*mBNj-b_aRSWg9u1j3EsQHg.png" /></figure><p>The division by 0.6745 is a normalisation factor that adjusts the MAD to be consistent with the standard deviation for normally distributed data. Specifically, it relates the MAD to the standard deviation of a normal distribution, where approximately 50% of the data lies within 0.6745 standard deviations from the median.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*Htm8jkR39zXWNMLwMg3sNA.png" /></figure><h3>Step 8: Compute Weights for Huber Loss</h3><p>Next, we compute weights based on the scaled residuals:</p><pre>weights = np.where(np.abs(scaled_residuals) &lt;= epsilon, 1, epsilon / np.abs(scaled_residuals))</pre><p><strong>Weights</strong>: Residuals less than epsilon get a weight of 1 (treated normally), while larger residuals are down-weighted, reducing their impact on the regression.</p><h3>Step 9: Weighted Least Squares Update</h3><p>Now we apply the weights to the features for the regression update:</p><pre>X_weighted = X_train_with_intercept * weights.reshape(-1, 1)  # Weighting the features</pre><p>This step modifies the feature set to reflect the weights derived from the residuals.</p><h3>Step 10: Regularisation</h3><p>We also add a regularisation term to avoid overfitting:</p><pre>regularization_matrix = alpha * np.eye(n_features)  # Regularization matrix<br>regularization_matrix[0, 0] = 0  # Exclude intercept from regularization<br><br># Coefficient update with regularization<br>coef_update = np.linalg.pinv(X_weighted.T @ X_weighted + regularization_matrix) @ (X_weighted.T @ y_train)</pre><h3>Step 11: Check for Convergence</h3><p>We check if the coefficients have converged:</p><pre>if np.linalg.norm(coef - coef_update) &lt; tol:<br>    print(&quot;break&quot;)<br><br>coef = coef_update</pre><h3>Step 12: Visualising Results</h3><p>Finally, we visualise the results of the Huber regression:</p><pre>plt.scatter(X_train, y_train, color=&#39;blue&#39;, label=&#39;Training Data&#39;)  # Training data<br>plt.plot(X_train, X_train @ coef[1:] + coef[0], color=&#39;red&#39;, linestyle=&#39;-&#39;)  # Fitted line<br>plt.title(&#39;Huber Regression with Intercept&#39;)<br>plt.xlabel(&#39;Carat&#39;)<br>plt.ylabel(&#39;Price&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*9STHNIjENtgMzRUaEg_tXA.png" /></figure><p>These results are only after 1 iteration. Huber implements multiple iterations on the dataset to reach to a convergence point thereby reducing the impact of outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*9qtMi3Qm8UXD-quDJEXKwQ.png" /></figure><h3>4. Where Huber Regression Shines: Real-World Applications 🌍</h3><p>Huber Regression is invaluable in finance, environmental sciences, and any field where data can have unexpected spikes. It’s especially useful in cases where we don’t want to ignore outliers entirely but instead adjust our model’s sensitivity to them.</p><h4>Practical Example: Stock Prices</h4><p>Imagine we’re analyzing stock prices. Large fluctuations often occur due to market events, but we don’t want these events to dominate our predictions. Huber Regression allows us to incorporate these “outlier” points while still capturing the overall trend.</p><h3>5. Pitfalls and Misconceptions of Huber Regression 🚧</h3><p>While Huber Regression is effective, it’s essential to choose epsilon carefully. An epsilon that’s too large will treat all points as inliers, resembling MSE, while too small an epsilon will make it similar to MAE. Additionally, Huber Regression might not perform well when outliers make up a large portion of the dataset, as the algorithm might lose the overall trend.</p><h3>6. Conclusion 🌟</h3><p>Huber Regression is a flexible and powerful tool, designed to be resilient against outliers while staying accurate for most data points. By understanding the math behind it and leveraging it properly, we can develop models that are robust, balanced, and better suited for complex datasets with both regular patterns and extreme values.</p><p>Give Huber Regression a try on your next dataset with outliers — let your model be sensitive to the overall trend, but don’t let outliers dominate the story!</p><p>If you prefer visual understanding, check out my 8 mins video link on the <a href="https://youtu.be/C6Z8Ps23Omw"><strong>Huber Regressor!</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7be42ef07cdd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use SGD Regressor When Data is Constantly Updating: A Comprehensive Guide to Stochastic Gradient…]]></title>
            <link>https://medium.com/@algostalk/use-sgd-regressor-when-data-is-constantly-updating-a-comprehensive-guide-to-stochastic-gradient-61f48fe3337c?source=rss-06c43f99303c------2</link>
            <guid isPermaLink="false">https://medium.com/p/61f48fe3337c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression-modeling]]></category>
            <category><![CDATA[stochastic-gradient]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[AlgoStalk]]></dc:creator>
            <pubDate>Tue, 29 Oct 2024 23:48:58 GMT</pubDate>
            <atom:updated>2024-10-29T23:48:58.440Z</atom:updated>
            <content:encoded><![CDATA[<h3>Use SGD Regressor When Data is Constantly Updating: A Comprehensive Guide to Stochastic Gradient Descent in Machine Learning</h3><p>If you prefer visual understanding, check out my video link on the <a href="https://youtu.be/OrlH53yb8Dw">SGD Regressor!</a></p><p>In machine learning, where data often updates continuously, choosing the right algorithm to handle this dynamic input is essential. One powerful technique is <strong>Stochastic Gradient Descent (SGD)</strong>, particularly through its application in the SGD Regressor. In this article, we’ll explore what makes SGD unique, how it handles real-time data effectively, and walk through its mathematical foundation in a step-by-step guide.</p><h3>Understanding the SGD Regressor</h3><p>The SGD Regressor is particularly valuable when working with data that updates frequently, as it adjusts to new information without requiring a full retraining process. But what is SGD, and how does it work? Let’s start with the basics of <strong>Gradient Descent</strong>.</p><h3>Gradient Descent: Finding the “Valley” of Minimum Error</h3><p>Imagine you’re standing on a hill and need to reach the lowest point. To do this, you’ll follow the steepest downhill path, adjusting your steps based on how steep each section of the hill is. In machine learning, this process — following the “gradient” or slope of the error curve — is called <strong>Gradient Descent</strong>. It’s our way of finding the lowest point, or minimum error, for our model.</p><ol><li><strong>The Slope</strong>: Think of the slope as the steepness of the hill. In machine learning, the slope of the error function tells us how much error we’re making with our current model parameters.</li><li><strong>Adjusting Weights</strong>: Each model parameter (or weight) can be fine-tuned. The gradient tells us how much to adjust each weight to reduce prediction errors.</li><li><strong>The Path to Minimum Loss</strong>: By iterating through these steps, we gradually approach the lowest possible error, improving model accuracy.</li></ol><h3>The Mathematical Concept of Gradient Descent</h3><p>For a basic linear regression model, the relationship between input features and output is given by:</p><p>y = mx + b</p><p>where:</p><ul><li>y is the predicted output,</li><li>x is the input feature,</li><li>m (or weight) adjusts the impact of xxx, and</li><li>b (or bias) shifts the prediction.</li></ul><p>The <strong>loss function</strong>, commonly Mean Squared Error (MSE), measures how far off our predictions are from actual values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*lEIE3A6TkqNAIubtpK50vA.png" /></figure><p>where:</p><ul><li>n is the number of samples,</li><li>y_actual​ is the actual output, and</li><li>y_predicted​ is the predicted output.</li></ul><p>The goal of gradient descent is to <strong>minimise this loss function</strong> by adjusting mmm and b in a way that reduces prediction error.</p><h3>What Makes SGD Different?</h3><p><strong>Stochastic Gradient Descent</strong> is a variant of gradient descent with two main differences:</p><ol><li><strong>One Step at a Time</strong>: Instead of calculating the gradient on the entire dataset, which is slow and computationally intense, SGD updates weights based on just one data point at a time. This is like adjusting your path after each step instead of waiting to gather data from the entire hill.</li><li><strong>Speed and Real-Time Adaptability</strong>: Because it only considers one data point, SGD learns faster and adapts well to real-time data streams.</li></ol><p>However, SGD updates introduce noise due to using individual data points. While this can make updates less stable, it sometimes helps the model escape local minima, potentially finding better solutions.</p><h3>Mathematical Update Rule in SGD</h3><p>In SGD, the weight update for each training example follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*pj7Kd6jhrasfWknUmK2xQA.png" /></figure><p>where eta(η) is the <strong>learning rate</strong> — a small number controlling the update size. A smaller learning rate ensures smoother updates, while a larger one speeds up learning but may overshoot the minimum.</p><h3>Step-by-Step Math Behind Gradient Descent</h3><p>Understanding the core mechanics of gradient descent requires examining the steps of error reduction:</p><ol><li><strong>Error Function</strong>: The error function measures how far off our predictions are. In simple linear regression, we typically use the Mean Squared Error:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*lEIE3A6TkqNAIubtpK50vA.png" /></figure><p><strong>2. Calculating the Gradient</strong>: The gradient provides the slope of the error function, guiding us on how to adjust each weight to reduce error.</p><ul><li><strong>Gradient for m​ (slope)</strong>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*zVAw5mfZAOd9V2T3OYCb1g.png" /></figure><p><strong>Example</strong>: If x = 10 and the error term (y_actual − y_predicted)=0.4:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*dqrEw2-TrRYNceVsSewXrw.png" /></figure><ul><li><strong>Gradient for b (intercept)</strong>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*0pJFZwbF8PtFnYdHlK6GSg.png" /></figure><ul><li>This calculation is similar, but you do not multiply by x.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VsoFa1g6eVhLgaQacgVBvg.png" /></figure><h3>Why and When to Use SGD Regressor</h3><h3>Ideal Scenarios for Using SGD Regressor:</h3><ol><li><strong>Large Datasets</strong>: SGD handles large datasets efficiently, as it doesn’t require loading the entire dataset at once.</li><li><strong>Real-Time Data</strong>: SGD’s ability to adapt to new data makes it suitable for dynamic environments, like stock prices or recommendation engines.</li><li><strong>High-Dimensional Data</strong>: SGD quickly adapts, reducing computational costs for datasets with many features, such as images or text.</li></ol><h3>Practical Use Cases:</h3><ul><li><strong>Dynamic Pricing</strong>: Constant updates to supply and demand data make SGD ideal for online platforms with dynamic pricing.</li><li><strong>Recommendation Systems</strong>: Real-time updates in user behavior data allow recommendation models to stay relevant without requiring full retraining.</li><li><strong>Financial Market Prediction</strong>: Quickly adapting to trends in financial data, SGD-based models offer agility in changing markets.</li></ul><h3>Advantages of Using the SGD Regressor</h3><ol><li><strong>Efficiency</strong>: Works well with large datasets and real-time data.</li><li><strong>Flexibility</strong>: Supports various loss functions like Mean Absolute Error (MAE).</li><li><strong>Simplicity</strong>: The concept is straightforward, making it easy to implement.</li></ol><h3>How SGD Regressor Compares to Linear Regression</h3><p>A traditional linear regression model requires the full dataset to update, which is computationally expensive. By contrast, SGD allows for quick adaptation with real-time data, making it an excellent choice for scenarios where data is continuously updating.</p><h3>Limitations of the SGD Regressor</h3><ol><li><strong>Sensitivity to Noise</strong>: SGD’s single-data-point updates can amplify noise, leading to less stable convergence.</li><li><strong>Tuning Required</strong>: Learning rate and other parameters need careful tuning to avoid overshooting the minimum or converging too slowly.</li><li><strong>Memory Constraints</strong>: While efficient with streaming data, SGD still needs careful memory management for high-dimensional datasets.</li></ol><h3>Practical Tips</h3><ol><li><strong>Learning Rate</strong>: Start with a small learning rate and adjust as needed for stable convergence.</li><li><strong>Batch Size</strong>: Mini-batches (small groups of data points) can smooth the gradient descent path, reducing noise.</li><li><strong>Regularization</strong>: Adding L1 or L2 regularization helps manage overfitting with noisy data.</li><li><strong>Early Stopping</strong>: Stop training when performance plateaus to save computational resources.</li></ol><h3>Conclusion</h3><p>The SGD Regressor is a powerful tool for machine learning, particularly in environments where data updates frequently. Whether you’re beginning in machine learning or refining your skills, understanding the SGD Regressor is essential for dynamic applications. Try experimenting with various learning rates, mini-batch sizes, and regularisation techniques to see how this versatile model adapts to different types of data.</p><p>If you prefer visual understanding, check out this link on the <a href="https://youtu.be/OrlH53yb8Dw">SGD Regressor!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61f48fe3337c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Linear Regression: How to Tackle Overfitting with Ridge, Lasso, and Elastic Net]]></title>
            <link>https://medium.com/@algostalk/mastering-linear-regression-how-to-tackle-overfitting-with-ridge-lasso-and-elastic-net-7da2c5ee9abc?source=rss-06c43f99303c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7da2c5ee9abc</guid>
            <category><![CDATA[ridge-regression]]></category>
            <category><![CDATA[lasso-regression]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[regularization]]></category>
            <dc:creator><![CDATA[AlgoStalk]]></dc:creator>
            <pubDate>Tue, 15 Oct 2024 23:26:22 GMT</pubDate>
            <atom:updated>2024-10-15T23:26:22.279Z</atom:updated>
            <content:encoded><![CDATA[<p>Mastering Linear Regression, Overfitting, and Regularisation: A Deep Dive</p><p>“In this article, we’ll explore linear regression, overfitting, and regularisation techniques like Ridge, Lasso, and Elastic Net. If you prefer a video explanation, check out my detailed video on this topic on my YouTube channel, <a href="https://youtu.be/qtO_XeLcoDY">AlgoStalk.</a>”</p><p>Linear regression is a foundational tool in data science, offering a straightforward way to understand relationships between variables. However, this simplicity can lead to challenges, particularly when the model becomes too tailored to the training data — a phenomenon known as overfitting. In this article, we’ll explore the core mathematics of linear regression, the pitfalls of overfitting, and how regularization techniques like Ridge, Lasso, and Elastic Net can help create a more balanced model.</p><h3>Linear Regression: The Fundamentals</h3><p>At its heart, linear regression aims to find a linear relationship between independent variables (features) and a dependent variable (target). The goal is to fit a line (or hyperplane) to the data that minimises the prediction error.</p><p>Model Representation:</p><p>For a given dataset with n observations and p features, linear regression is modelled as:</p><pre>y = β₀ + β₁x₁ + β₂x₂ + ... + βₚxₚ + ε</pre><pre>where<br>- y is the target variable.<br>- β₀ is the intercept.<br>- β₀, β₁,..., βₚ are the coefficients.<br>- x₁, x₂,..., xₚ are the feature values.<br>- ϵ is the error term (or residuals).</pre><h3>Loss Function: Mean Squared Error (MSE)</h3><p>To determine the best-fitting line, linear regression uses a loss function to quantify how well the model’s predictions match the actual data. The most common loss function is the Mean Squared Error (MSE):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*5kRL56p42GH7ya-P" /></figure><p>Goal: Minimise the MSE by finding the optimal coefficients that produce the smallest error.</p><h3>Normal Equation: Finding the Optimal Coefficients</h3><p>For linear regression, we can find the coefficients directly using the Normal Equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*wfetvShM-O6Y3l-b" /></figure><p><strong>Normal Equation</strong></p><p>The Normal Equation provides a closed-form solution that minimises the MSE. However, it can be computationally expensive when the number of features is large or when X^TX is not invertible.</p><h3>The Challenge of Overfitting</h3><p>Overfitting occurs when the linear regression model learns the training data too well, capturing not just the underlying trend but also the noise. This results in low error on the training set but poor generalisation to new data, indicated by a high error on the test set.</p><h3>RMSE: Evaluating Model Performance</h3><p>To evaluate overfitting, we use the Root Mean Squared Error (RMSE), which gives a clear sense of the prediction error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*PrZhnAkGPlPfiFX-" /><figcaption>Root Mean Squared Error</figcaption></figure><p>Root Mean Squared Error</p><ul><li>Low training RMSE with high test RMSE suggests overfitting.</li><li>Balanced RMSE values between training and test sets indicate a well-generalised model.</li></ul><h3>Regularisation: A Solution to Overfitting</h3><p>Regularisation introduces a penalty to the loss function, discouraging the model from learning overly complex relationships. This helps in balancing the model’s fit, especially when some features have a strong but spurious influence on the target variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Cs-CF_9-II0jJ_DOQiXjQ.png" /><figcaption>Regularisation Techniques</figcaption></figure><h3>1. Ridge Regression (L2 Regularisation)</h3><p>Mathematics:</p><p>Ridge Regression modifies the linear regression loss function by adding an L2 penalty — the sum of the squared coefficients:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*uUE-NupgHjR7JzAe" /></figure><p>Where:</p><ul><li>λ is the regularisation parameter controlling the penalty strength.</li><li>A higher λ value increases the penalty, shrinking the coefficients more.</li><li>A lower λ allows the model to remain closer to simple linear regression.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AE5jb4uwsFc0HuEaka6k1g.png" /></figure><p>The closed-form solution for Ridge Regression is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*72sOhOPpGl24N7zO" /><figcaption>Normal Equation for Ridge</figcaption></figure><p>Effect: Ridge Regression does not eliminate features but reduces the size of coefficients, making them smaller and more balanced, which helps in handling multicollinearity.</p><h3>2. Lasso Regression (L1 Regularisation)</h3><p>Mathematics:</p><p>Lasso Regression adds an L1 penalty — the sum of the absolute values of the coefficients:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*m66sYcdeDvbawjIz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-aVNFi0LMrdxQjs2ZguvIg.png" /></figure><p>Key Difference: Lasso can shrink some coefficients to zero, effectively performing feature selection by eliminating less important features.</p><p>Effect: Lasso is beneficial when you have many irrelevant features, as it simplifies the model by retaining only the most impactful variables.</p><h3>3. Elastic Net: Combining the Best of Both</h3><p>Elastic Net combines the penalties of both Ridge and Lasso:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*3jRsT8AfgqJnerj2" /></figure><p>Where:</p><ul><li>λ1 controls the Lasso penalty.</li><li>λ2 controls the Ridge penalty.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*-JMMDk4CADYYIeds9BLjIg.png" /></figure><p>Effect: Elastic Net is particularly useful when dealing with highly correlated features or when you want to strike a balance between feature selection and managing large coefficients.</p><h3>Visualisation of Regularisation Effects</h3><p>Visualising how different values of λ affect Ridge and Lasso can help you understand their impact:</p><ul><li>As λ increases in Ridge, the coefficients are gradually reduced but remain non-zero.</li><li>In Lasso, increasing λ can push some coefficients to zero, removing less important features from the model.</li></ul><h3>Choosing the Right Regularisation Method</h3><ul><li>Ridge Regression: Suitable when all features contribute to the target and you want to reduce their influence rather than eliminate any.</li><li>Lasso Regression: Ideal when you suspect that only a few features are truly important and want a simpler model.</li><li>Elastic Net: Best when dealing with complex datasets with highly correlated features, offering flexibility to balance between Ridge and Lasso’s strengths.</li></ul><h3>Conclusion: Leveraging Regularisation for Better Models</h3><p>Regularisation provides a powerful way to prevent overfitting, ensuring that your linear regression models are not only accurate but also generalise well to new data. By adjusting the penalty term (λ) and selecting the appropriate regularization technique, you can find the sweet spot between simplicity and accuracy.</p><p>The choice of regularisation depends on your specific data and goals:</p><ul><li>Ridge helps in cases of multicollinearity without feature elimination.</li><li>Lasso simplifies the model by zeroing out irrelevant coefficients.</li><li>Elastic Net offers a balanced approach for complex, feature-rich datasets.</li></ul><p>With these mathematical tools in your arsenal, you can build more robust and interpretable models, turning data insights into actionable knowledge.</p><p>I hope you found this breakdown helpful. For a more visual explanation of these concepts, watch my YouTube video on <a href="https://youtu.be/qtO_XeLcoDY">Linear Regression and Regularization</a> on the AlgoStalk channel. Don’t forget to like and subscribe if you want more content like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7da2c5ee9abc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>