<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ameen Siddiqui on Medium]]></title>
        <description><![CDATA[Stories by Ameen Siddiqui on Medium]]></description>
        <link>https://medium.com/@asidd24?source=rss-6e0470393cdb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O8vNIcwvqo3Sx2kDfi-S1A.jpeg</url>
            <title>Stories by Ameen Siddiqui on Medium</title>
            <link>https://medium.com/@asidd24?source=rss-6e0470393cdb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:06:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@asidd24/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Datetime & Categorical Features: The Last Mile of Time Series Feature Engineering]]></title>
            <link>https://medium.com/@asidd24/datetime-categorical-features-the-last-mile-of-time-series-feature-engineering-f401c00c6a79?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f401c00c6a79</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 06:18:30 GMT</pubDate>
            <atom:updated>2026-10-06T06:18:30.799Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Part 10 of 10 · Time Series, Engineered</strong></p><p><em>Holiday windows had twice the error of everything else. Then I gave the model the calendar — and the whole series came together in one feature matrix.</em></p><p>Here is the number that reorganised how I think about the calendar.</p><p>On a synthetic panel of 200 stores, a LightGBM model with lags, rolling windows and seasonality scored <strong>9.55% WMAPE</strong> overall. Inside the ±3-day windows around holidays, the same model scored <strong>16.98%</strong>.</p><p>Nearly <strong>twice the error, concentrated in a handful of days a year.</strong></p><p>Adding holiday and payday features took the overall score to <strong>6.68%</strong> — a 30% reduction — and the holiday windows to 9.33%.</p><p>This is the last article of the series. Nine parts covered the <em>dynamics</em> (lags, windows, trend) and the <em>shape</em> (decomposition, seasonality) of a series. This one covers the last mile: <strong>events the calendar knows about, categories the panel carries, and how to assemble all of it into one matrix that does not leak.</strong></p><p>Companion code: <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></p><blockquote><strong>A note on the data.</strong> Everything below runs on a <strong>synthetic</strong> panel: 200 stores × 1,096 days (2022–2024), train on 2022–23, test on 2024, 7-day-ahead direct forecasts, LightGBM, WMAPE. The “Black Friday” in it is simply the 4th Friday of November with a ramp-up and spike that I designed — not the real retail event. Synthetic data lets me know the true noise floor. It also means the effects are <em>by construction</em> present. Treat the numbers as a demonstration of mechanism, not a forecast of your own gains.</blockquote><h3>Part 1 — Events Are Not Seasonality</h3><p>Article 9 encoded cycles that repeat on a fixed period. Events do not.</p><ul><li>Easter moves by up to five weeks.</li><li>“4th Friday of November” lands on a different day-of-year every year.</li><li>Paydays follow the business-day calendar, not the date.</li><li>Demand does not just <em>spike</em> on the day — it ramps up before and dips after.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w26xnPmpWlumsSXobpotzg.png" /><figcaption><strong>[FIGURE 1 — Event Effects]</strong> <em>Average demand multiplier around each event: ramp-up before, spike on the day, dip after. Dates shift year to year.</em></figcaption></figure><p>Fourier terms tied to day-of-year can only memorise “something happened around Nov 22–28 in the years I saw.” The moment the event moves, they are wrong.</p><p>An event feature says it directly: <em>this day is, or is near, the event.</em></p><h3>Part 2 — The Event Feature Toolkit</h3><p>Four ingredients cover most of what I have needed in retail and demand work:</p><pre>import pandas as pd, numpy as np<br><br>def add_event_features(df, holidays):<br>    &quot;&quot;&quot;holidays: DataFrame with columns [date, name]; known in advance.&quot;&quot;&quot;<br>    d = df.copy()<br>    d[&#39;is_holiday&#39;] = d[&#39;date&#39;].isin(holidays[&#39;date&#39;]).astype(int)<br><br>    h = np.sort(holidays[&#39;date&#39;].values)<br>    dates = d[&#39;date&#39;].values<br>    # signed distance to the next / since the previous event<br>    nxt = np.searchsorted(h, dates, side=&#39;left&#39;)<br>    prv = np.searchsorted(h, dates, side=&#39;right&#39;) - 1<br>    d[&#39;days_to_holiday&#39;]   = np.where(nxt &lt; len(h), (h[np.minimum(nxt, len(h)-1)] - dates) / np.timedelta64(1, &#39;D&#39;), 60)<br>    d[&#39;days_since_holiday&#39;] = np.where(prv &gt;= 0, (dates - h[np.maximum(prv, 0)]) / np.timedelta64(1, &#39;D&#39;), 60)<br>    d[[&#39;days_to_holiday&#39;, &#39;days_since_holiday&#39;]] = d[[&#39;days_to_holiday&#39;, &#39;days_since_holiday&#39;]].clip(upper=60)<br><br>    # payday &amp; business-day structure<br>    d[&#39;is_month_end_bday&#39;] = d[&#39;date&#39;].eq(d[&#39;date&#39;] + pd.offsets.BMonthEnd(0)).astype(int)<br>    d[&#39;bdays_to_month_end&#39;] = (d[&#39;date&#39;] + pd.offsets.BMonthEnd(0)).sub(d[&#39;date&#39;]).dt.days<br>    return d</pre><ol><li><strong>Flag</strong> — is_holiday. The cheapest and biggest first step.</li><li><strong>Countdown / count-up</strong> — days <em>to</em> the next event and days <em>since</em> the last. These let a tree learn the ramp-up and the hangover, which a flag cannot.</li><li><strong>Payday and month-end structure</strong> — last business day, mid-month payday, business days remaining.</li><li><strong>Named events</strong> — if events behave differently (Christmas Eve vs. Easter), pass the event type as a categorical, not just a flag.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zK9bi-AH7JrES_tF4TTZMA.png" /><figcaption><strong>[FIGURE 2 — Event Feature Gains]</strong> <em>WMAPE on all test days vs. holiday windows (±3 days). Error concentrates where the events are, and so does the fix.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tV5ghbkhXGitopo6cYRttA.png" /></figure><p>Read the last column carefully. The flag captured most of the gain. Countdowns improved the windows further. <strong>Paydays improved the overall score and nudged the holiday windows slightly the wrong way (9.07 → 9.33)</strong> — they help on the days that are not holidays. A single metric hides that; always slice the error by event type.</p><h3>Part 3 — Categorical Features in a Panel</h3><p>Panels bring a column that single series never had: store_id, sku, region. With 200 levels, the question is how to hand it to a tree model. I compared six options, with and without lag features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SryYIfiCFpOqNw-yvZbrQ.png" /><figcaption><strong>[FIGURE 3 — Categorical Encodings]</strong> <em>Left: with lags present, every encoding of store_id lands in a 0.13-point band. Right: with lags removed (a cold-start setting), the choice matters enormously.</em></figcaption></figure><p><strong>With lags</strong> (the normal case): none 6.68%, integer 6.75%, one-hot 6.69%, LightGBM native 6.68%, target encoding 6.77–6.81%. <strong>Nothing mattered.</strong> The lag features already carry each store’s level, so store identity is redundant — the same way Article 9 showed day-of-week dummies were redundant next to lag_7.</p><p><strong>Without lags</strong> (new store, new SKU, no history): none <strong>35.37%</strong>, integer 11.63%, one-hot 11.97%, native 11.30%, naive target encoding 10.71%, <strong>expanding target encoding 8.02%</strong>.</p><p>Two lessons:</p><ul><li>Categorical encoding earns its place where history cannot speak for the entity: <strong>cold start, sparse histories, new products.</strong></li><li>Expanding target encoding won by a distance here — but I want to be careful. It does not only encode <em>identity</em>; the running mean also carries a time-varying level. Some of its win is that it behaves like a slow lag feature. Credit it accordingly.</li></ul><p>My practical default: <strong>LightGBM native categorical for ≤ a few hundred levels, target encoding when levels are very high-cardinality or history is thin — and one-hot only when the cardinality is small.</strong></p><h3>Part 4 — Target Encoding Is a Leakage Machine</h3><p>Target encoding replaces a category with the mean of the target for that category. That makes it a <strong>fitted feature</strong> — and fitted features need the discipline from Article 3: compute them using only information available at prediction time.</p><p>Three variants:</p><pre># 1. NAIVE — the category mean includes the row&#39;s own target. Leaks.<br>df[&#39;te&#39;] = df.groupby(&#39;store&#39;)[&#39;y&#39;].transform(&#39;mean&#39;)<br><br># 2. EXPANDING — only strictly earlier rows. Leak-safe in time.<br>df[&#39;te&#39;] = (df.groupby(&#39;store&#39;)[&#39;y&#39;]<br>              .transform(lambda s: s.shift(1).expanding().mean()))<br><br># 3. FIT-ON-ALL — computed before the train/validation split. Validation leaks.</pre><p>I ran a controlled experiment, varying the number of training rows per category level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8rxg2NOL37shvvc3aZV7YA.png" /><figcaption><strong>[FIGURE 4 — Target Encoding Leakage]</strong> <em>Left: naive encoding flatters the training fit, most when each level has few rows. Right: fit the encoding on all rows before splitting and validation error drops below the noise floor — which is impossible.</em></figcaption></figure><p>What the data showed, honestly:</p><ul><li><strong>Train/held-out gap.</strong> At 4 rows per level, naive encoding trained to a MAE of 6.83 but scored 9.06 on held-out data. At 128 rows per level the gap closes (7.96 vs 8.00). The fewer rows per level, the more the model “sees” its own answer.</li><li><strong>Held-out error was not hurt much by naive encoding.</strong> In this test, naive and leak-safe encodings reached similar held-out MAE. I expected a bigger penalty and did not find one — so I will not claim it. The damage of naive encoding is mainly that <strong>it makes your validation misleading and your model needlessly unstable at small n.</strong></li><li><strong>The real hazard is fitting the encoding before the split.</strong> Validation MAE fell to <strong>7.82 at n=4 — below the irreducible noise floor of 7.98.</strong> No honest model can beat the noise floor. That is a leak, and it is the cleanest <em>diagnostic</em> in the article:</li></ul><blockquote>If your validation error is below the noise you can estimate, you have leaked — not discovered something.</blockquote><p><strong>Rules for target encoding in time series:</strong></p><ol><li>Use <strong>expanding</strong> (shifted) statistics in time, or <strong>out-of-fold</strong> encodings with time-respecting folds.</li><li><strong>Refit inside every CV fold.</strong> Never compute it once on the full frame.</li><li>Add <strong>smoothing</strong> for sparse levels (blend toward the global mean).</li><li>Prefer LightGBM’s native categorical when you can — it avoids this class of bug.</li></ol><h3>Part 5 — One Feature Matrix, With a Leakage Ledger</h3><p>Nine articles produce nine feature families. Assembling them is where leaks sneak in, because each family has a different rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NXPxyuhlN5ivW_eG1-llpA.png" /><figcaption><strong>[FIGURE 5 — The Leakage Ledger]</strong> <em>Every block in the final matrix and the rule that keeps it safe: shift ≥ horizon, known in advance, or expanding / out-of-fold only.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IN5ckc4ldfSnmrxQ1P3G_g.png" /></figure><p>The whole matrix came to <strong>35 columns</strong>. The test for any column is one question: <em>at forecast time, for a date 30 days out, can I fill this value?</em> If yes, it is safe. If it needs a target value you do not have yet, it must be shifted or fitted per fold.</p><h3>Part 6 — The Capstone Ablation (and a Surprise)</h3><p>I added the blocks one at a time and measured the 2024 WMAPE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2FqEtL3zAKyON1z3f2COcA.png" /><figcaption><strong>[FIGURE 6 — Capstone Ablation]</strong> <em>Cumulative WMAPE as each block is added. Bars are all test days; diamonds are holiday windows.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNYztg2RgdQwckWHkpByMA.png" /></figure><p>Three things I want you to take from this table:</p><p><strong>1. The big wins are lags and holidays.</strong> Lags took the error from 10.58% to 8.33%. Holidays took it from 9.55% to 6.92%. Everything else is close to noise.</p><p><strong>2. Two blocks made this panel <em>worse</em>.</strong> Adding t_index moved the error up from 8.31% to 8.79%, and adding Article 9&#39;s Fourier seasonality moved it up again to 9.55%. This is not what I wrote in Article 9 — and it is not a contradiction. Article 9&#39;s experiment had a single series with a strong yearly cycle and no moving events. Here, my hypothesis (supported by a month-by-month breakdown, where the extra error sat in the months with moving one-off events) is that <strong>date-tied Fourier terms memorise the moving events, and the lags already carry the level, so </strong><strong>t_index is redundant.</strong> Once the explicit holiday block arrived, the damage vanished. I did not test this fully; take it as a hypothesis worth checking on your data.</p><p>The lesson: <strong>a feature block that helps in isolation can hurt in combination.</strong> Add blocks one at a time, in a fixed order, and keep the table.</p><p><strong>3. The store column added nothing.</strong> With lags in the matrix, 6.68% → 6.68%. Cold-start is where it earns its place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xD8CD-8hNgeSHG6XXaJ1lw.png" /></figure><h3>The Leakage Rules for This Article</h3><p><strong>Rule 1 — Events must be known in advance.</strong> A promotion flag back-filled after the fact is a future leak in a calendar costume. Test: can you fill this column 30 days ahead?</p><p><strong>Rule 2 — Target encoding is a fitted feature.</strong> Expanding or out-of-fold only, refit per fold.</p><p><strong>Rule 3 — Build deterministic calendar blocks on the full frame; refit data-estimated blocks per fold.</strong> Fourier terms and holiday flags are functions of the date. Group means of the target are not.</p><p><strong>Rule 4 — Slice the error.</strong> Overall WMAPE hid that paydays hurt the holiday windows.</p><h3>Key Takeaways</h3><ul><li><strong>Error concentrates on events.</strong> Holiday windows ran 16.98% vs. 9.55% overall; event features cut the overall error by ~30% (9.55% → 6.68%)</li><li><strong>Moving events need event features,</strong> not day-of-year Fourier terms</li><li>A <strong>flag</strong> gets most of the gain; <strong>countdown / count-up</strong> gets the ramp and the hangover</li><li><strong>Categoricals matter when history cannot speak</strong> — cold start went 35.4% → 8.0% with expanding target encoding; with lags present the choice made no real difference (6.68–6.81%)</li><li><strong>Target encoding is a fitted feature.</strong> Fitting it before the split pushed validation below the noise floor — the diagnostic for a leak</li><li><strong>Blocks that help alone can hurt together.</strong> t_index and Fourier seasonality made this panel worse until holiday features arrived</li><li>Every block in the final 35-column matrix has a <strong>leakage rule</strong>; the ledger is the checklist</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YSa4eroPtAEwbnwugqVbVQ.png" /></figure><p>Ten articles, one thread: <strong>the model is rarely the bottleneck. The features — and the discipline about when each one is known — are.</strong></p><h3>Try This Week</h3><p>Take a model you run in production. Slice its error by (a) holiday windows and (b) days since last payday. If the error is double on those slices, the calendar is your next feature block. Then check one more thing: is any fitted feature computed before your validation split?</p><blockquote>If you run it, I would like to hear what you found.</blockquote><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></blockquote><p><em>Thank you for reading all ten parts. If this series saved you a bug or sharpened a decision, share it with one person on your team who works with time series.</em></p><p>#TimeSeries #Forecasting #FeatureEngineering #MachineLearning #DataScience #Python #LightGBM #TargetEncoding</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f401c00c6a79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seasonality Features: Dummies, Fourier Terms & Beyond]]></title>
            <link>https://medium.com/@asidd24/seasonality-features-dummies-fourier-terms-beyond-e0e4df1e83b0?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0e4df1e83b0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[forecasting]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-09-29T03:31:01.723Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article 9 of 10 in the series: Feature Engineering for Time Series Forecasting</em></p><blockquote><strong>Most forecasting models encode seasonality wrong.</strong> They drop a month integer column into the feature matrix and move on. That column tells the model December and January are eleven units apart — the single largest distance on the calendar — when they are in fact adjacent. In the benchmark below, adding integer calendar columns made the model <strong>worse than using no calendar features at all.</strong> Encoding seasonality is not a formality. It is a modelling decision with a measurable cost.</blockquote><h3>Seasonality Is Never One Cycle</h3><p>Articles 6, 7 and 8 covered the three feature families that handle <em>dynamics</em>: lag features for sequential memory, window features for local statistics, trend features for unbounded growth. All three learn from the <strong>values</strong> of the series.</p><p>Seasonality is different. It is learned from <strong>where you are in the calendar</strong> — and a real business series is almost never sitting on one calendar cycle at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nS4m_941WHO78DNY3u3bZQ.png" /><figcaption><strong>[FIGURE 1 — Multiple Seasonalities]</strong> <em>Three years of daily sales. Top: the raw series (blue) with a 28-day centred mean (black) that strips out the weekly cycle and exposes the annual one. Bottom left: the weekly profile — a sharp, irregular shape with a Friday/Saturday trading peak and a Sunday collapse. Bottom right: the yearly profile — a smooth annual swing with a distinct December spike. These two cycles are running simultaneously, on completely different timescales, with completely different shapes.</em></figcaption></figure><p>That difference in <em>shape</em> is the whole article. The weekly profile is a jagged seven-point pattern — Sunday is 24 units below the weekly mean while Saturday is 18 above, and there is no smooth curve connecting them. The yearly profile is a smooth wave with one sharp holiday event.</p><p><strong>One encoding cannot serve both.</strong> The rest of this article is about picking the right one for each.</p><blockquote><strong>A note on the data.</strong> Every figure and number here comes from a synthetic daily series with a <em>deliberately non-sinusoidal</em> weekly effect and a yearly effect built from a smooth wave plus a Gaussian holiday peak. The series generator in this repo, generate_daily_sales, uses pure sine waves for both — which would quietly rig this comparison, since a single sin/cos pair reproduces a pure sine exactly. Real seasonality is not a sine wave, so the benchmark series here isn&#39;t either. Full generation code is in make_figures.py alongside this article.</blockquote><h3>The Ordinal Trap</h3><p>Here is the most common seasonality bug in production forecasting code, and it is one line long:</p><pre>df[&#39;month&#39;] = df.index.month        # 1, 2, 3, ..., 12<br>df[&#39;dayofweek&#39;] = df.index.dayofweek  # 0, 1, 2, ..., 6</pre><p>These look like features. They are actually <strong>assertions about distance</strong> — and the assertions are false.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqPR8K98UJH_zkcv5FJ64g.png" /><figcaption><strong>[FIGURE 2 — The Ordinal Trap]</strong> <em>Left: with an integer column, December (12) and January (1) sit at opposite ends of the range. The model is told they are 11 units apart, while June and July are 1 unit apart. Right: the sin/cos encoding maps the calendar onto a circle. December→January and June→July are now both exactly 0.518 apart. The calendar closes.</em></figcaption></figure><p>The cyclical fix is two columns per cycle:</p><pre>from src.features import add_cyclical_encoding<br><br>df = add_cyclical_encoding(df, &#39;month&#39;, period=12)      # month_sin, month_cos<br>df = add_cyclical_encoding(df, &#39;dayofweek&#39;, period=7)   # dayofweek_sin, dayofweek_cos</pre><p>sin(2πx/period) and cos(2πx/period) together place each calendar position on a unit circle, so adjacency is preserved everywhere — including across the year boundary.</p><p><strong>This is the standard advice, and it is only half right.</strong> Cyclical encoding fixes the <em>distance</em> problem. It does not fix the <em>shape</em> problem, and for short cycles the shape problem is the bigger one.</p><h3>Encoding 1 — Dummies, for Short Cycles with Irregular Shapes</h3><p>A one-hot dummy per calendar position makes no assumption about shape at all. Each position gets its own free parameter:</p><pre>dow_dummies = pd.get_dummies(df[&#39;dayofweek&#39;], prefix=&#39;dow&#39;).astype(int)<br>df = pd.concat([df, dow_dummies], axis=1)</pre><p>Seven columns for day-of-week (six free parameters once you have an intercept). Each day is free to take whatever value the data says, with no requirement that neighbouring days be similar.</p><p>Compare that against a single sin/cos pair on the real weekly shape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ub4c-6f9y9wlQv7ErUgPzQ.png" /><figcaption><strong>[FIGURE 3 — Dummies vs Cyclical vs Fourier]</strong> <em>Recovering the true weekly effect (black) three ways. Day-of-week dummies (green) reproduce it almost exactly — maximum error 0.56. A single sin/cos pair (red) cannot: it is one smooth wave being asked to fit a shape with a Friday/Saturday peak and a Sunday cliff, and its maximum error is 19.62 — roughly 35× worse. Three Fourier pairs at period 7 (orange) match the dummies exactly, for the same reason explained below.</em></figcaption></figure><p>The single sin/cos pair fails badly here, and it fails <strong>structurally, not because of tuning</strong>. Two parameters describe exactly one sine wave of a fixed period — one peak, one trough, smooth everywhere. The true weekly pattern has a sharp discontinuity between Saturday and Sunday. No amplitude or phase can produce that.</p><p><strong>Note the orange line.</strong> Three Fourier pairs at period 7 recover the shape as perfectly as the dummies do — because 3 pairs is 6 parameters, exactly the 6 free parameters of 7 dummies. At short periods the two encodings are the same model written differently. Dummies are simply the more readable spelling, so use those.</p><p><strong>Rule:</strong> for a short cycle (period ≤ 12) where you have many observations per position, use dummies. Day-of-week on daily data, month on monthly data, hour on hourly data.</p><h3>Encoding 2 — Fourier Terms, for Long Cycles</h3><p>Now take the yearly cycle on daily data: period 365.25, and note it is <em>not an integer</em>, which alone disqualifies dummy encoding.</p><p>Fourier terms encode a cycle as a sum of sine/cosine pairs at increasing frequencies:</p><pre>from src.features import add_fourier_terms<br><br># 6 pairs = 12 columns, encoding yearly seasonality on daily data<br>df = add_fourier_terms(df, period=365.25, n_terms=6, prefix=&#39;fy&#39;)</pre><p>The parameter that matters is n_terms (K). It controls how much seasonal detail the encoding can represent:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dEiW1bateFzDuszMFbmsMQ.png" /><figcaption><strong>[FIGURE 4 — Fourier Terms at K = 1, 2, 3, 6, 10]</strong> <em>The same yearly effect (black) approximated with increasing numbers of Fourier pairs. K=1 (2 columns) is a single wave and misses the December peak entirely — RMSE 5.74. K=3 gets the broad annual swing but still flattens the holiday spike — RMSE 4.30. K=6 (12 columns) captures the peak — RMSE 1.87. K=10 is closest to the true curve at RMSE 1.00, but note the ringing either side of the sharp peak: that oscillation is the encoding starting to describe features that aren’t there.</em></figcaption></figure><p>Each additional pair adds a higher-frequency harmonic, so the encoding can represent progressively sharper features. The trade-off is the familiar one: <strong>K is a smoothness dial.</strong> Too low and you flatten real seasonal events; too high and you start fitting noise, plus the Gibbs-style ringing visible around day 345 in the K=10 curve.</p><p><strong>My default is K=6 for yearly seasonality on daily data</strong>, then tune from there — down to 3 if the annual shape is a simple wave, up to 10 if there are genuine sharp seasonal events (holidays, fiscal year ends, school terms). For weekly seasonality on daily data, don’t use Fourier at all; use the seven dummies.</p><p>One caveat on Figure 4’s RMSE numbers: they measure fit against the <em>true</em> seasonal shape, which we only know because this series is synthetic. On real data you cannot compute that column, which is why the next two figures measure held-out error instead.</p><h3>Why Not Just Use Dummies for the Yearly Cycle Too?</h3><p>Because there are 365 of them, and you do not have enough data.</p><p>With three years of daily history, a day-of-year dummy encoding gives each parameter roughly <strong>two or three observations</strong> to estimate from. A parameter fit on two noisy observations is a parameter fit on noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Az9t6CqgOpVHCRCFv6W2Vw.png" /><figcaption><strong>[FIGURE 5 — Dummy Sparsity vs Fourier]</strong> <em>Both panels fit the yearly component on the same two years of training data and are evaluated on a held-out third year. Left: 365 day-of-year dummies (red) produce a jagged curve that tracks the training noise rather than the seasonal signal — held-out RMSE 7.21. Right: 6 Fourier pairs (green), just 12 parameters estimated from all 730 training observations, recover the smooth shape — held-out RMSE 6.44.</em></figcaption></figure><p>The irreducible noise floor in this series is 6.00 (the standard deviation of the noise term). Against that floor:</p><ul><li>6 Fourier pairs land at 6.44 — about <strong>7% above the best achievable error</strong></li><li>365 dummies land at 7.21 — about <strong>20% above it</strong>, nearly three times the excess error</li></ul><p>And the Fourier encoding achieves that with <strong>12 columns instead of 365.</strong></p><p>This is the general principle: <em>dummies spend one parameter per calendar position, Fourier terms spend a fixed budget across the whole cycle.</em> When positions are plentiful and observations per position are scarce — which is exactly what a long period means — the fixed budget wins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmtFO3_zxHWC-b6IMXJAzQ.png" /></figure><h3>The Head-to-Head: Does Any of This Actually Move Accuracy?</h3><p>Everything above is about representation quality. Here is what it is worth in forecast error.</p><p>The setup: LightGBM, 7-day-ahead forecasting (every feature is known at least 7 days before its target), trained on two years and evaluated on a held-out third year. Every configuration shares the same base features — lags 7/14/21/28, rolling mean and std over 7 and 28 days, and a linear t_index trend column from Article 8. The <strong>only</strong> thing that changes is the seasonal encoding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dKiZxFgqWaS8O_vSnfp8YQ.png" /><figcaption><strong>[FIGURE 6 — Encoding Comparison (WMAPE)]</strong> <em>WMAPE on the held-out year, with each configuration’s change against the no-calendar baseline. Day-of-week dummies plus 6 yearly Fourier pairs is best at 4.25% (+31.7%). Yearly Fourier terms alone get almost all of that at 4.30% (+31.0%). Day-of-week dummies alone reach 5.97% (+4.2%). Cyclical sin/cos lands marginally worse than baseline (6.26% vs 6.23%, −0.5%), and integer calendar columns are the worst configuration tested at 6.36% (−2.1%) — genuinely worse than including no calendar features at all.</em></figcaption></figure><p>Three findings worth sitting with, in increasing order of usefulness.</p><p><strong>1. Integer encoding is not merely suboptimal — it is actively harmful.</strong> At 6.36% WMAPE it finished below the baseline with no calendar features whatsoever (6.23%). The false distance metric gives the tree splits that look informative on training data and don’t generalise. If you take one thing from this article: deleting a raw month integer column is sometimes a free accuracy win.</p><p><strong>2. Cyclical sin/cos was nearly worthless here (6.26% vs the 6.23% baseline — marginally worse).</strong> This surprises people, because sin/cos encoding is the most-repeated seasonality advice on the internet. But Figure 3 already told us why: one sin/cos pair per cycle cannot represent this weekly shape, and one pair at period 12 cannot represent the December spike. The encoding fixes adjacency and then has no capacity left to do anything useful. <strong>Cyclical encoding is a fix for the ordinal trap, not a seasonality feature in its own right.</strong></p><p><strong>3. The yearly Fourier terms delivered nearly all the gain — 6.23% → 4.30%, a 31% error reduction — while day-of-week dummies delivered only 6.23% → 5.97%, a 4% reduction.</strong> This asymmetry is the most practically important result here, and the reason is worth understanding.</p><h3>Why the Weekly Dummies Underperformed (And What It Tells You)</h3><p>The base feature set includes sales_lag_7, sales_lag_14, sales_lag_21 and sales_lag_28 — all multiples of 7.</p><p><strong>A lag at a multiple of the seasonal period is an implicit seasonal feature.</strong> sales_lag_7 is the same weekday, one week ago. The model already has a strong signal about the weekly pattern before you add a single dummy column. The dummies are largely redundant, and their 4% improvement is the small residual value they add on top.</p><p>Nothing in the base feature set does that for the yearly cycle. The longest lag is 28 days. To capture annual seasonality through lags alone you would need lag_365 — which costs you a full year of training data to the resulting NaNs, and is noisy besides, since it leans on one single observation from last year.</p><p><strong>The operating rule this gives you:</strong></p><blockquote>Seasonal features earn their place in proportion to how <em>far</em> their period sits beyond your longest lag. Short-period seasonality is mostly already in your lag features. Long-period seasonality is not reachable by lags at all, and explicit encoding is the only way the model ever sees it.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WA9lrjZVVIam4ijdspTiTA.png" /></figure><p>The practical progression for a daily business series:</p><ol><li>Day-of-week dummies (7 columns) — cheap, safe, usually a small win</li><li>Yearly Fourier terms at K=6 (12 columns) — usually the big win</li><li>Tune K by held-out error: try 3, 6, 10</li><li>Add month dummies only if a month-boundary effect exists that the yearly Fourier terms miss</li><li>Delete every raw integer calendar column you find</li></ol><p>Nineteen columns total. That is the entire seasonal feature set for most daily series, and it assembles in about ten lines:</p><pre>import pandas as pd<br>from src.features import add_datetime_features, add_fourier_terms<br><br>def add_seasonal_features(df: pd.DataFrame, yearly_k: int = 6) -&gt; pd.DataFrame:<br>    &quot;&quot;&quot;<br>    The complete seasonal feature set for a daily series — 19 columns.<br><br>    Build this on the FULL contiguous frame, before any train/test split<br>    (see Rule 1 below). Requires a sorted DatetimeIndex.<br><br>    add_datetime_features and add_fourier_terms are thin wrappers in the<br>    companion repo — the first extracts df.index.dayofweek into a column,<br>    the second appends 2*K sin/cos columns for the given period.<br>    &quot;&quot;&quot;<br>    df = add_datetime_features(df, features=(&#39;dayofweek&#39;,))<br><br>    # Short cycle -&gt; 7 dummies, free to fit an irregular weekly shape<br>    dow = pd.get_dummies(df[&#39;dayofweek&#39;], prefix=&#39;dow&#39;).astype(int)<br>    df = pd.concat([df.drop(columns=&#39;dayofweek&#39;), dow], axis=1)<br><br>    # Long cycle -&gt; 12 columns, smooth and safe at a non-integer period<br>    df = add_fourier_terms(df, period=365.25, n_terms=yearly_k, prefix=&#39;fy&#39;)<br><br>    return df<br><br>seasonal = add_seasonal_features(sales.to_frame())<br># dow_0 ... dow_6, fy_sin_1 ... fy_cos_6  -&gt;  19 seasonal columns</pre><p>Note the df.drop(columns=&#39;dayofweek&#39;): the integer column is scaffolding used to build the dummies, and it leaves with them. Per Figure 6, keeping it around is not neutral — it costs accuracy.</p><p>Bolt this onto the lag, window and trend features from Articles 6–8 and the feature matrix is complete.</p><h3>The Leakage Rules for Seasonality Features</h3><p>Good news first: <strong>this is the one feature family in the series that is inherently leakage-free.</strong> Day-of-week, month, and Fourier terms are deterministic functions of the calendar. They use no observed values of the target, so computing them over the full series before splitting leaks nothing. No refitting per fold, unlike the STL components in Articles 3–5 or the trend fits in Article 8.</p><p>Three caveats, though, and each has bitten real pipelines:</p><p><strong>Rule 1 — Build Fourier terms on the full contiguous frame, before splitting.</strong> add_fourier_terms indexes off t = np.arange(len(df)) starting at the first row. Build them separately on a train frame and a test frame and the test terms restart at phase zero, silently misaligning every seasonal column. Build once, split after. This is a correctness bug, not a leakage bug, and it is much easier to ship than you would think.</p><p><strong>Rule 2 — Seasonal indices <em>estimated from data</em> are not calendar features.</strong> A day-of-week multiplier computed as a group mean over the target is fitted from observations, so it follows the normal walk-forward discipline: refit inside each fold. The rule only exempts encodings that are pure functions of the timestamp.</p><p><strong>Rule 3 — Holiday and event flags are only safe if the calendar is known in advance.</strong> Fixed-date holidays are fine. A “promotion was running” flag, a “competitor stockout” flag, or anything back-filled after the fact is a future-information leak wearing a calendar costume. The test is simple: at forecast time, for a date 30 days out, can you fill this column? If not, it is not a calendar feature.</p><h3>Key Takeaways</h3><ul><li>Real series carry <strong>multiple seasonal cycles at once</strong>, with different periods <em>and</em> different shapes — one encoding will not serve them all</li><li><strong>Raw integer calendar columns assert false distances</strong> (December is 11 units from January) and measured <em>worse than no calendar features at all</em> in the benchmark — 6.36% vs 6.23% WMAPE</li><li><strong>Cyclical sin/cos fixes adjacency, not shape.</strong> One pair is one smooth wave; it cannot represent a sharp weekly profile and delivered almost nothing over baseline here</li><li><strong>Dummies</strong> are right for short periods with many observations per position — they reproduced the irregular weekly effect with 35× less error than a sin/cos pair</li><li><strong>Fourier terms</strong> are right for long periods: 12 columns beat 365 day-of-year dummies on held-out error (6.44 vs 7.21 RMSE against a 6.00 noise floor), and they handle non-integer periods like 365.25 that dummies structurally cannot</li><li><strong>K is a smoothness dial</strong> — K=6 is a good default for yearly-on-daily; too high adds ringing around sharp seasonal events</li><li>The biggest single win was <strong>yearly Fourier terms: 31% WMAPE reduction.</strong> Day-of-week dummies gave 4%, because lags at multiples of 7 already carried most of the weekly signal</li><li><strong>Seasonal value scales with distance beyond your longest lag</strong> — encode the cycles your lag features cannot reach</li><li>Calendar encodings are <strong>leakage-free by construction</strong>, but build Fourier terms before splitting, and treat data-estimated seasonal indices and back-filled event flags as ordinary fitted features</li></ul><h3>What’s Next</h3><p>Article 10 — <em>“Datetime &amp; Categorical Features: The Last Mile of Time Series FE”</em> — closes the series. Calendar features beyond the seasonal cycle (holidays, paydays, business-day counts, time-to-event countdowns), categorical encoding for multi-series panels, and how to combine everything from Articles 1–9 into a single feature matrix that survives contact with production.</p><p><strong>→ Subscribe to the newsletter to get Article 10 next Tuesday.</strong></p><p><em>If you have a </em><em>month integer column sitting in a production feature matrix right now, run the experiment: drop it, retrain, compare. Reply with your before/after numbers if you&#39;re willing to share — I&#39;m building a follow-up piece on how often this actually helps and would love ~10 real examples from practitioners.</em></p><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></blockquote><p>#TimeSeries #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #LightGBM #Seasonality #FourierTerms</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0e4df1e83b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trend Features: Making Tree-Based Models Extrapolate]]></title>
            <link>https://medium.com/@asidd24/trend-features-making-tree-based-models-extrapolate-11af78cc2435?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/11af78cc2435</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-09-22T03:31:01.942Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article 8 of 10 in the series: Feature Engineering for Time Series Forecasting</em></p><blockquote><strong><em>Tree-based models cannot extrapolate.</em></strong><em> If your test set is at a higher trend level than your training set, LightGBM will flatline at the last value it saw. This is not a bug. It is the mathematical property of how decision trees make predictions — and it silently destroys most forecasting models the moment they hit real production data.</em></blockquote><h3>The Failure Mode Most Tutorials Ignore</h3><p>Every article in this series so far has been about squeezing more signal out of a time series — lag features, window features, decomposition, imputation, anomaly detection. Everything assumes that if you feed a tree-based model enough well-engineered features, it will produce good forecasts.</p><p>For strongly trending data, that assumption is wrong.</p><p>Not “sometimes wrong” or “wrong in edge cases.” Wrong in a specific, predictable, catastrophic way that affects every real-world business time series with meaningful growth or decay: <strong>retail demand growing year-over-year, energy consumption climbing with a customer base, sensor readings drifting as equipment ages, user engagement decaying as a product matures.</strong></p><p>Here’s what it looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZuGIaADhTXlmO2HT9Cmow.png" /><figcaption>Figure 1 — Same features, same training data, same forecast horizon. LightGBM (red) uses lag features and rolling statistics. It flatlines at ~168 while the actual series (blue) continues climbing to ~200. Linear regression (green dashed) on the identical features extrapolates correctly. The difference isn’t tuning. It’s the fundamental way trees make predictions vs the way linear models do.</figcaption></figure><p>Look at Figure 1 carefully. The LightGBM prediction is a <strong>flat horizontal line</strong> near the top of the training range. Not a bad forecast — a <em>categorically wrong</em> one. In production, this looks like your demand forecast systematically under-predicts for months, planners lose trust in the model, and the model gets replaced with someone’s Excel projection.</p><p>This article covers the three feature classes that fix it.</p><h3>Why Trees Can’t Extrapolate</h3><p>A decision tree makes predictions by traversing a series of splits and returning the mean of the training values in the terminal leaf. That process has a critical constraint: <strong>the tree can only predict values it has seen during training.</strong></p><p>If your training data ranges from 100 to 170, your tree’s predictions will always fall between 100 and 170. Every terminal leaf holds a mean of training values. Every mean of training values is bounded by the range of training values.</p><p>This is fundamentally different from linear models. A linear regression finds the slope and intercept that best fit training data, then extrapolates that relationship forward without limit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oHEHWU_k3eB44I0xBkSqeg.png" /><figcaption>Figure 2 — Two-panel diagram. Left: within the training range, the tree (red step function) tracks the data acceptably. Right: beyond the training range, the tree’s prediction flatlines (structurally cannot exceed the training range), while linear regression continues extrapolating. This constraint is inherent to decision trees; no amount of hyperparameter tuning fixes it.</figcaption></figure><p><strong>The critical implication for time series:</strong> if your target is trending upward, and you use time-related features (lag_1, rolling_mean_28), those features themselves stay within the historical range. The tree sees “lag_1 = 165, roll_mean_28 = 160” and predicts something inside the training range because that’s the only vocabulary it has.</p><p>Random Forests, XGBoost, LightGBM, CatBoost — <strong>all tree-based methods have this constraint.</strong> Deep learning models with unbounded outputs can extrapolate; classical statistical models (ARIMA, ETS) can extrapolate; but the workhorses of modern tabular ML cannot.</p><p>The fix is to add features that inherently <em>grow with time</em> — features whose value at prediction time is outside the training range because time itself has moved forward.</p><h3>Feature Class 1 — The Linear Time Index</h3><p>The simplest and most effective trend feature is a linear counter — a monotonically increasing integer starting at 0.</p><pre>import numpy as np<br>import pandas as pd<br><br>df[&#39;t_index&#39;] = np.arange(len(df))</pre><p>That’s it. One column. At training time, the model sees t_index ranging from 0 to N. At prediction time, t_index takes values greater than N — outside the training range.</p><p>Why does this fix the extrapolation problem? Because the model learns that t_index correlates with the target, and when t_index extends beyond the training range at prediction time, the tree&#39;s splits force the prediction into the highest terminal leaf (which contains the highest training values). Combined with lag features, this recovers most of the missing trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EhL3Y-HtwsCbUXOV16eqGg.png" /><figcaption>Figure 3 — Same LightGBM model. Same lag features. Same training data. Only difference: a single t_index column added to the feature matrix. The dashed red line (without t_index) flatlines. The solid green line (with t_index) tracks the trend correctly through the entire test period. One feature. Massive impact.</figcaption></figure><p><strong>Two things to understand about the fix:</strong></p><p><strong>It’s not a true extrapolation.</strong> The tree is still bounded — but now the highest terminal leaf corresponds to t_index = N, and predictions in that leaf are the mean of the highest-trend training observations. When t_index exceeds N, predictions stay at that leaf&#39;s mean. This is why the green line tracks the trend but doesn&#39;t perfectly match slope — the tree is stepping between plateaus of training-range values, not continuously extrapolating.</p><p><strong>The fix works best when the trend is roughly linear.</strong> If your series accelerates or decelerates, a single linear counter can be misleading. That’s what Feature Classes 2 and 3 address.</p><h3>Feature Class 2 — Polynomial Trend Features</h3><p>When your trend curves — accelerating growth, decelerating decay, S-curve maturity — a linear time index under-captures the shape. Polynomial features extend the idea:</p><pre>df[&#39;t_index&#39;]   = np.arange(len(df))<br>df[&#39;t_squared&#39;] = df[&#39;t_index&#39;] ** 2<br>df[&#39;t_cubed&#39;]   = df[&#39;t_index&#39;] ** 3</pre><p>Each polynomial term captures a different aspect of curvature. t_squared handles quadratic acceleration, t_cubed handles S-curves and inflection points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzSjNdRiOvlXoBO4AOETQQ.png" /><figcaption>Figure 4 — Polynomial fits at degree 1 (green), degree 2 (yellow), and degree 5 (red) on a mildly curving series. Within the training range, higher-degree polynomials fit better. Beyond the training range, they extrapolate wildly — degree 5 shoots off dramatically. This is why polynomial trend features carry real risk in production.</figcaption></figure><p><strong>The trade-off is severe.</strong> Every additional polynomial degree gives you more flexibility inside the training range and more extrapolation risk outside it. In Figure 4, the degree-5 polynomial fits the training data slightly better than degree 2, but its forecast in the test period is much worse.</p><p><strong>My rule in practice:</strong> never go beyond degree 2 for a raw polynomial trend feature unless you have overwhelming evidence the underlying process is genuinely cubic or higher. And even at degree 2, monitor the model closely for the first forecast horizon in production — polynomial trends can turn on you fast.</p><p>If you find yourself reaching for degree 3+, you almost certainly want piecewise trends instead.</p><h3>Feature Class 3 — Piecewise Linear Trends and Changepoints</h3><p>Real business time series rarely have a single trend rate across all history. Growth accelerates when a product goes viral, decelerates when the market saturates, breaks when a competitor enters, resets when a pandemic hits.</p><p>Piecewise linear trends model this by fitting a <em>sequence</em> of linear trends across the series, with <strong>changepoints</strong> marking where the slope changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKz4y-7HupYoslLNhvxOyQ.png" /><figcaption>igure 5 — A series with three genuine trend regimes: slow growth, rapid growth, then plateau. Two changepoints mark the regime transitions. Fitting a single linear trend across all data would average these three regimes together, producing a slope that matches none of them. Piecewise trends respect the actual structure.</figcaption></figure><p>The implementation pattern:</p><pre>def add_piecewise_trend(df, changepoints):<br>    &quot;&quot;&quot;<br>    Add a linear time index plus knots at each changepoint.<br>    Each knot activates only after its changepoint.<br>    &quot;&quot;&quot;<br>    df = df.copy()<br>    t = np.arange(len(df))<br>    df[&#39;t_index&#39;] = t<br>    for cp in changepoints:<br>        df[f&#39;trend_after_{cp}&#39;] = np.maximum(0, t - cp)<br>    return df<br><br># Detected or specified changepoints<br>changepoints = [200, 400]<br>df = add_piecewise_trend(df, changepoints)</pre><p>The np.maximum(0, t - cp) construction creates a feature that is zero before the changepoint and grows linearly after — letting the model add an extra slope contribution for the post-changepoint period.</p><p><strong>Where do changepoints come from?</strong></p><ul><li><strong>Business knowledge</strong> — you know a product launched on date X, or a pandemic hit on date Y</li><li><strong>Detection algorithms</strong> — the ruptures library, PELT algorithm, or Prophet&#39;s automatic changepoint detection</li><li><strong>Domain-driven candidates</strong> — quarter boundaries, fiscal year transitions, marketing campaign starts</li></ul><p>Prophet’s changepoint detection is the pragmatic default. It fits many candidate changepoints, then regularizes most of them toward zero, keeping only the ones the data supports. If you need automatic changepoint discovery and don’t want to hand-tune, fitting Prophet purely to extract its detected changepoints (then using them in your LightGBM pipeline) is a legitimate workflow I’ve used in production.</p><h3>The Pattern That Wins in Production — Detrend and Refit</h3><p>The strongest pattern I’ve deployed for trending series isn’t a single feature at all. It’s a workflow: <strong>fit a linear trend on training data, subtract it from the series, forecast the detrended residuals with your tree-based model, then re-add the trend for the final forecast.</strong></p><p>The steps:</p><ol><li>Fit a linear model to the training portion of the series</li><li>Subtract the linear trend from the whole series (both train and test periods)</li><li>Train your tree-based model on the detrended series — it now only needs to learn seasonality and short-term dynamics</li><li>At forecast time, predict the detrended value, then add back the extrapolated linear trend.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FRRo72n3gis24o1RsOS5KA.png" /><figcaption>Figure 6 — The hybrid pattern in three steps. Top: fit linear trend on training data (solid green), extrapolate to test period (dashed green). Middle: detrend the series — now stationary, so the tree only needs to learn seasonality and noise. Bottom: forecast the detrended series with LightGBM, add back the extrapolated trend. Result: green forecast tracks blue actuals across the entire test period.</figcaption></figure><p>Code:</p><pre>from sklearn.linear_model import LinearRegression<br>import lightgbm as lgb<br><br># Step 1 — Fit linear trend on training portion<br>t_arr = np.arange(len(sales))<br>train_end = int(len(sales) * 0.7)<br><br>lin = LinearRegression().fit(<br>    t_arr[:train_end].reshape(-1, 1),<br>    sales.values[:train_end]<br>)<br>trend_line = lin.predict(t_arr.reshape(-1, 1))<br><br># Step 2 — Detrend<br>detrended = sales.values - trend_line<br><br># Step 3 — Forecast detrended series with LightGBM<br># (build lag features on detrended, train, predict as usual)<br># ... standard pipeline ...<br><br># Step 4 — Re-add trend to final forecast<br>final_forecast = detrended_forecast + trend_line[test_indices]</pre><p><strong>Why this works better than a single time_index feature:</strong></p><ul><li>The tree model only sees a stationary series — no trend to model, no extrapolation problem</li><li>Feature engineering on stationary data is more effective; seasonal patterns are cleaner to detect</li><li>The linear trend component handles extrapolation naturally, using linear regression’s proper extrapolation math</li><li>The two components are fitted independently, so you can tune each without interference</li></ul><p>This is roughly the pattern Prophet uses internally (trend component + seasonal component + holidays), except you get to pick your own tree-based model for the residual dynamics — which usually beats Prophet’s residual model on any dataset with reasonable feature engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-gzgitOW987h2CgwZW4P-w.png" /></figure><p>For most business time series with clear growth or decay, the practical progression is:</p><ol><li>Start with t_index alone</li><li>If residual analysis shows the trend is curved, add t_squared</li><li>If the model still misses trend changes, switch to piecewise with 1–3 changepoints</li><li>If accuracy matters and the trend is strong, use the hybrid detrend pattern</li></ol><p>The hybrid pattern is my default for anything at Evonik-scale (annual demand forecasting where trend contribution dominates seasonal contribution). For smaller projects with less severe trend, t_index alone usually gets 80% of the benefit at 5% of the complexity.</p><h3>The Leakage Rules for Trend Features</h3><p>Same discipline as every other feature in this series, with one addition specific to trend features:</p><p><strong>Rule 1 — The linear trend fit for the hybrid pattern must be fit on training data only.</strong> Never fit the linear trend on the full series (train + test), then use it in cross-validation. That leaks the future trend into training.</p><p><strong>Rule 2 — For walk-forward CV, refit the trend at each fold origin.</strong> The trend fit in fold 1 (using data through month 6) is different from the trend fit in fold 5 (using data through month 30). Every fold gets its own trend.</p><p><strong>Rule 3 — Piecewise changepoints from detection algorithms must be detected on training data only.</strong> Prophet-detected changepoints or ruptures-detected changepoints computed on the full series leak future regime information into training.</p><p>Skip these and your walk-forward CV will show falsely encouraging numbers, then production accuracy will disappoint. This is the same class of subtle leakage bug we covered in Articles 3 and 4 with STL, applied to trend components.</p><h3>Key Takeaways</h3><ul><li>Tree-based models (LightGBM, XGBoost, RandomForest, CatBoost) <strong>cannot extrapolate</strong> — predictions are structurally bounded by the training range</li><li>For trending series, this means the model <strong>flatlines</strong> at test time — a categorical failure, not a tuning problem</li><li><strong>Linear time index</strong> (t_index) is the simplest and most effective fix — one column, massive impact</li><li><strong>Polynomial trend features</strong> work for curved trends but carry real extrapolation risk beyond degree 2</li><li><strong>Piecewise linear trends with changepoints</strong> handle regime changes properly; use Prophet or ruptures for automatic detection</li><li>The <strong>hybrid detrend-and-refit pattern</strong> is the production-grade approach for strong-trend series — fit linear trend, subtract, forecast residuals, re-add trend</li><li>All trend feature fits must <strong>respect walk-forward CV boundaries</strong> — same leakage discipline as everywhere else in this series</li></ul><h3>What’s Next</h3><p>Article 9 — <em>“Seasonality Features: Dummies, Fourier Terms &amp; Beyond”</em> — completes the trio started in Articles 6–8. Lag features handle sequential memory, window features handle local statistics, trend features handle unbounded growth. Article 9 covers explicit seasonal encoding — day-of-week dummies, month indicators, Fourier terms for multiple seasonalities, and the specific patterns that make each appropriate.</p><p><strong>→ Subscribe to the newsletter to get Article 9 next Tuesday.</strong></p><p>If you’ve shipped a tree-based forecasting model in production, drop a comment on how you handled the extrapolation problem — or, more honestly, when you first noticed it. The moment of discovery is usually a demand planning meeting where someone points at a flatlined forecast and asks what happened. Every practitioner has that story.</p><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></blockquote><p>#TimeSeries #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #LightGBM #Prophet #Extrapolation</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11af78cc2435" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Window Features: Rolling, Expanding & Exponential Smoothing]]></title>
            <link>https://medium.com/@asidd24/window-features-rolling-expanding-exponential-smoothing-b8470fd0bbbf?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8470fd0bbbf</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 07:04:24 GMT</pubDate>
            <atom:updated>2026-09-08T07:04:24.135Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article 7 of 10 in the series: Feature Engineering for Time Series Forecasting</em></p><blockquote>A lag feature answers “what happened at time t-7?” A window feature answers “what did the last 7 days look like as a whole?” The distinction seems small until you see what each contributes to a tree-based model — where window features earn their place capturing something lag features fundamentally cannot: volatility and shape.</blockquote><h3>Why Window Features Exist</h3><p>Article 6 established lag features — the primary way tree-based models learn from sequential history. If lag features are so central to time series ML, why do window features exist at all?</p><p>Because a lag feature is a single point observation. It tells your model <em>“here’s what happened 7 days ago.”</em> A window feature is an aggregate statistic. It tells your model <em>“here’s what happened across the last 7 days as a whole — the average, the volatility, the peak, the trough.”</em></p><p>These are fundamentally different signals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZTyrtAUfOk3JH26nHgZbTw.png" /><figcaption>Figure 1 — Same series, same current observation. Top: lag_7 is a single point value — the sales exactly 7 days ago. Bottom: rolling_mean_7 is an aggregate summary of the last 7 days — the average level. Both features are computed from past data, but they encode completely different information.</figcaption></figure><p>The pedagogical case for window features rests on three things they capture that lags miss:</p><ol><li><strong>Local level</strong> — the recent average, smoothed of noise</li><li><strong>Local volatility</strong> — how much the series has been oscillating</li><li><strong>Local shape</strong> — the min, max, skewness of recent history</li></ol><p>Every business time series with meaningful downstream decisions benefits from at least the first two. Demand forecasting cares about <em>where the level is trending</em> (rolling mean) and <em>how confident we can be about it</em> (rolling std). Anomaly detection cares about whether <em>current volatility</em> matches <em>historical volatility</em>. Financial forecasting cares about <em>momentum vs mean reversion signatures</em> — differences between rolling means at multiple window sizes.</p><h3>Rolling Statistics — The Workhorse</h3><p>The most-used window feature is the rolling mean. In pandas:</p><pre># ❌ WRONG: at time t, the window includes y(t) itself<br>df[&#39;rolling_mean_7&#39;] = series.rolling(7).mean()<br><br># ✅ CORRECT: shift(1) before rolling ensures no leakage<br>df[&#39;rolling_mean_7&#39;] = series.shift(1).rolling(7).mean()</pre><p>The shift(1) before .rolling() is not optional. Without it, the rolling mean at time t includes y(t) — the very value your model is trying to predict. This is the single most common leakage bug in window features. Same rule as everywhere in this series: any statistic used to predict y(t) must be computed from strictly earlier values.</p><p>Rolling means at different window sizes capture different characteristics of the series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FFYAcFSrvBAN89-xy2OhdA.png" /><figcaption>Figure 2 — Rolling means at 7, 14, 28, and 90 day windows on the same series. Short windows (7 days) track the series closely — following weekly seasonality. Long windows (90 days) smooth into a slow-moving trend estimate. In production, including multiple window sizes lets the model reason about both short-term momentum and longer-term trend simultaneously.</figcaption></figure><p><strong>The rule I follow for choosing window sizes:</strong></p><ul><li>Include a rolling window at your <strong>seasonal period</strong> (7 for daily/weekly data, 12 for monthly/yearly)</li><li>Add one <strong>multiple of that</strong> (14 or 28 for weekly seasonality)</li><li>Optionally include a <strong>quarterly window</strong> (~90 days) for long-term trend</li></ul><p>Three windows total is usually enough. More adds noise faster than it adds signal.</p><h3>Beyond the Mean — Which Statistics to Include</h3><p>Rolling mean is the obvious default, but it’s not the most informative window feature for a tree model. In practice, <strong>rolling standard deviation earns its keep on almost every problem</strong> because it captures something the mean cannot: volatility.</p><pre>s = series.shift(1)  # apply once, use everywhere<br><br>df[&#39;roll_mean_7&#39;]  = s.rolling(7).mean()<br>df[&#39;roll_std_7&#39;]   = s.rolling(7).std()<br>df[&#39;roll_min_7&#39;]   = s.rolling(7).min()<br>df[&#39;roll_max_7&#39;]   = s.rolling(7).max()</pre><p><strong>The finding that surprises most practitioners:</strong> rolling std is often one of the top-3 features in a demand or operational forecasting model. Not because volatility directly predicts the next value, but because tree-based models can learn interactions like <em>“when volatility is high AND the level is dropping, predict a further drop.”</em> That interaction is invisible to lag features alone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qvSNPltOcz2NDocQGXlcbQ.png" /><figcaption>Figure 3 — A series where volatility structurally increases at the midpoint. The top panel shows the raw series — the change is visually noticeable but easy to miss. The bottom panel shows rolling standard deviation with two window sizes. The short window (7-day, orange) is noisy but reacts fast. The long window (28-day, purple) is stable but slower to fully accept the new regime. Including both gives a tree model access to “recent volatility” as a feature — enabling downstream decisions to be conditional on the current regime.</figcaption></figure><p><strong>n production forecasting:</strong></p><ul><li>Rolling <strong>mean</strong> captures level</li><li>Rolling <strong>std</strong> captures volatility (regime shifts, volatility clustering)</li><li>Rolling <strong>min/max</strong> capture extremes and range</li><li>Rolling <strong>skew</strong> captures asymmetry (useful for demand data with occasional spikes)</li></ul><p>For most forecasting problems, mean + std is enough. Add min/max if your series has meaningful extremes (returns, sensor peaks, demand spikes). Add skew rarely — it’s often noisy at short windows.</p><h3>Expanding Windows — The Long-Run Reference</h3><p>An expanding window grows over time. At time t, it computes the statistic over <em>all history up to </em><em>t-1</em>, rather than a fixed window.</p><pre># Expanding mean = long-run average of all past values<br>df[&#39;exp_mean&#39;] = series.shift(1).expanding(min_periods=7).mean()<br>df[&#39;exp_std&#39;]  = series.shift(1).expanding(min_periods=7).std()</pre><p>The min_periods=7 parameter ensures the statistic is only computed once you have at least 7 observations — prevents NaN early in the series from becoming a single-value mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*szD6ybI7DWLoZkiuxruR-A.png" /><figcaption>Figure 4 — Rolling mean (red) tracks recent behavior — it moves as the series moves. Expanding mean (green) is the long-run average of all history — it moves slowly and reflects the overall trajectory. Both are useful features; they encode different information.</figcaption></figure><p><strong>When to include expanding features:</strong></p><ul><li>The long-run mean is a <strong>legitimate reference point</strong> for the series (mature businesses with a stable underlying process)</li><li>You want the model to have access to a <strong>regime-invariant baseline</strong> it can compare recent observations against</li><li>Your series has a <strong>long history</strong> — expanding features are much more informative on 5 years of data than on 5 months</li></ul><p><strong>When expanding features are the wrong choice:</strong></p><ul><li>The series has undergone <strong>structural regime changes</strong> that make old data irrelevant. Expanding statistics will slowly incorporate obsolete history, dragging the reference point toward values that no longer represent normal behavior.</li><li>The series is <strong>young</strong> — for less than a full seasonal cycle of history, expanding statistics are basically an artifact of your training data start point.</li></ul><p>In practice, I include expanding mean and expanding std as low-priority features. Tree-based models sometimes use them, sometimes ignore them entirely. The cost of including them is low — one column each, no meaningful computation overhead.</p><h3>Exponentially-Weighted Moving Averages — The Best of Both</h3><p>EWM sits between rolling windows and expanding windows. Instead of giving equal weight to every value in a window (rolling) or equal weight to every value ever seen (expanding), EWM gives <strong>exponentially decaying weight to older values</strong>.</p><p>Recent observations dominate. Older observations still contribute — just less.</p><pre># EWM mean with a specific &quot;span&quot; parameter<br>df[&#39;ewm_7&#39;]  = series.shift(1).ewm(span=7, adjust=False).mean()<br>df[&#39;ewm_30&#39;] = series.shift(1).ewm(span=30, adjust=False).mean()</pre><p>The span parameter is the natural way to specify how quickly the weights decay. A span of 7 means the &quot;effective window&quot; is roughly 7 observations — the most recent 7 carry the bulk of the weight. A span of 30 spreads weight over roughly a month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HWC-xTeoxauj0AKWelmCFg.png" /><figcaption>Figure 5 — EWM at spans of 7, 30, and 90. Short spans track recent behavior aggressively. Long spans smooth into a stable trend estimate. The advantage over rolling means: EWM has no hard window boundary, so it doesn’t “forget” older values abruptly — it just weights them less. This makes it smoother and less sensitive to individual points falling in and out of the window.</figcaption></figure><p><strong>EWM vs Rolling Mean — when to use which:</strong></p><ul><li><strong>Rolling mean</strong> — when you want a strict window of recent observations, and specifically want older data completely excluded</li><li><strong>EWM</strong> — when you want a smoother estimate that still respects the recency of information, without the arbitrary hard cutoff of a rolling window</li></ul><p>In my production pipelines at Evonik, I’ve found EWM features consistently outperform rolling mean at equivalent window sizes for macroeconomic indicators — where a single-month spike shouldn’t completely leave the model’s memory after N months. For clean operational data with less noise, rolling mean is usually preferred because it&#39;s more interpretable.</p><p><strong>The </strong><strong>adjust=False parameter matters.</strong> Pandas defaults to adjust=True which uses a different weighting scheme that isn&#39;t standard for time series feature engineering. Always specify adjust=False unless you have a specific reason not to — it matches how statsmodels and most forecasting libraries handle EWM.</p><h3>What LightGBM Actually Values</h3><p>Let me stop asserting and show. Fit a LightGBM model on a feature matrix containing lag features, rolling stats, expanding stats, and EWM. Look at what the model prioritizes.</p><pre>import lightgbm as lgb<br><br># Build the feature matrix (all leakage-safe with shift(1))<br>df = pd.DataFrame({&#39;target&#39;: sales})<br><br>for lag in [1, 7, 14, 28]:<br>    df[f&#39;lag_{lag}&#39;] = sales.shift(lag)<br><br>s = sales.shift(1)<br>for w in [7, 28]:<br>    df[f&#39;roll_mean_{w}&#39;] = s.rolling(w).mean()<br>    df[f&#39;roll_std_{w}&#39;]  = s.rolling(w).std()<br>    df[f&#39;roll_min_{w}&#39;]  = s.rolling(w).min()<br>    df[f&#39;roll_max_{w}&#39;]  = s.rolling(w).max()<br><br>df[&#39;exp_mean&#39;] = s.expanding(min_periods=7).mean()<br>df[&#39;exp_std&#39;]  = s.expanding(min_periods=7).std()<br>df[&#39;ewm_7&#39;]    = s.ewm(span=7, adjust=False).mean()<br>df[&#39;ewm_30&#39;]   = s.ewm(span=30, adjust=False).mean()<br><br>model = lgb.LGBMRegressor(n_estimators=200, learning_rate=0.05, max_depth=5)<br>model.fit(df.drop(columns=&#39;target&#39;), df[&#39;target&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XbNfcMyuWWxgBBGmn2SBaA.png" /><figcaption>Figure 6 — LightGBM feature importance across all four categories of features on a weekly-seasonal series with a volatility regime change. Seasonal lags (7, 14, 28) dominate the top of the chart — consistent with Article 6’s finding. But rolling std, EWM, and rolling mean at short windows also rank highly. Expanding features rank near the bottom for this series — the underlying process has changed enough that the long-run average is less informative than recent statistics.</figcaption></figure><p>Three observations that generalize across most business time series.</p><p><strong>Seasonal lags still win overall.</strong> For any strongly seasonal series, the exact-cycle lags dominate. This is the pattern from Article 6, and it holds after adding window features.</p><p><strong>Rolling std and EWM sneak into the top tier.</strong> Neither is a lag feature, and both add information the lag features cannot provide. Rolling std at 7 days ranks 5th overall — capturing recent volatility that lags cannot express. EWM at span=7 ranks 4th overall — providing a smoother reference than a hard 7-day window.</p><p><strong>Expanding features rank low.</strong> This series had a volatility regime change midway through, so long-run statistics are less informative than recent ones. On a stable series with a well-defined long-run average, expanding features would rank higher. The lesson: window feature type has to match the underlying dynamics of the series.</p><h3>The Leakage Bugs Specific to Window Features</h3><p>Article 2 covered the general leakage patterns. Window features have two additional traps that catch practitioners.</p><h3>Trap 1 — Forgetting the shift(1)</h3><pre># ❌ Rolling window at time t includes y(t)<br>df[&#39;rolling_mean_7&#39;] = series.rolling(7).mean()<br><br># ✅ Shift by 1 first<br>df[&#39;rolling_mean_7&#39;] = series.shift(1).rolling(7).mean()</pre><p>Trivial one-character fix. Every practitioner has done this at least once. Every practitioner regrets it.</p><h3>Trap 2 — Window size larger than the forecast gap</h3><p>If you’re forecasting y(t+12) and using rolling_mean_7 computed at time t, you&#39;re technically not leaking (the window at t sees y(t-7) through y(t-1)), but you&#39;re also silently including near-future information relative to the training target.</p><p>The safer pattern for multi-step forecasting is to increase the shift to match your horizon:</p><pre># Forecasting horizon = 12<br>horizon = 12<br><br># Any rolling window feature must be shifted by at least horizon<br># to ensure the window doesn&#39;t leak forward-looking information<br>df[&#39;rolling_mean_7&#39;] = series.shift(horizon).rolling(7).mean()</pre><p>This is how the reusable add_rolling_features in the <a href="https://github.com/Ameen2488/time-series-feature-engineering/blob/main/src/features.py">companion repo</a> handles it — the min_horizon parameter enforces this shift explicitly. Skip it and your model scores well in CV then degrades sharply in production because it was quietly using features that included values 1-11 steps into what would be the forecast window.</p><h3>Decision Framework</h3><p>Not every window feature belongs in every pipeline. Here’s the practical framework I use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyJZoYtDHaEv4IKDwWgRcQ.png" /></figure><p>A minimal defensible feature set for a demand forecasting model:</p><pre># 8 window features, each earning its place<br>df[&#39;roll_mean_7&#39;]  = s.rolling(7).mean()<br>df[&#39;roll_mean_28&#39;] = s.rolling(28).mean()<br>df[&#39;roll_std_7&#39;]   = s.rolling(7).std()<br>df[&#39;roll_std_28&#39;]  = s.rolling(28).std()<br>df[&#39;roll_max_7&#39;]   = s.rolling(7).max()<br>df[&#39;roll_min_7&#39;]   = s.rolling(7).min()<br>df[&#39;ewm_7&#39;]        = s.ewm(span=7,  adjust=False).mean()<br>df[&#39;ewm_30&#39;]       = s.ewm(span=30, adjust=False).mean()</pre><p>Combined with 4–6 lag features (Article 6), this gives you ~14 total features — enough for LightGBM to build a strong model, few enough that every feature has a defensible role.</p><h3>Key Takeaways</h3><ul><li>Window features capture what lag features cannot: <strong>local level, volatility, and shape</strong> of recent history</li><li><strong>Rolling mean</strong> at seasonal-period windows is the default; <strong>rolling std</strong> earns its place on almost every problem by capturing volatility</li><li><strong>Expanding features</strong> are good for mature series with stable long-run averages; poor for series with regime changes</li><li><strong>EWM</strong> sits between rolling and expanding — smoother than rolling, more responsive than expanding</li><li>Every window feature must be preceded by .shift(1) at minimum (or .shift(horizon) for multi-step forecasting) to prevent leakage</li><li>Feature importance in real models: <strong>seasonal lags dominate, but rolling std and EWM at short windows consistently rank in the top tier</strong></li></ul><h3>What’s Next</h3><p>Article 8 — <em>“Trend Features: Making Tree-Based Models Extrapolate”</em> — addresses a specific weakness of tree-based models that’s mostly ignored in tutorials: they cannot extrapolate beyond values seen in training. If your test set is at a higher trend level than your training set, LightGBM will systematically under-predict. Article 8 covers the three classes of trend features (linear time index, polynomial, piecewise) that let tree models handle this correctly.</p><p><strong>→ Subscribe to the newsletter to get Article 8 next Tuesday.</strong></p><p>If you’ve built a forecasting model in production, drop a comment on which window features you converged on and which categories (mean, std, min/max, EWM) you actually kept in the final model. The pruning decisions are always the most interesting part.</p><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></blockquote><p>#TimeSeries #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #LightGBM #RollingWindow #EWM</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8470fd0bbbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lag Features: Teaching Your Model to Remember the Past]]></title>
            <link>https://medium.com/data-science-collective/lag-features-teaching-your-model-to-remember-the-past-005a0493ed32?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/005a0493ed32</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Wed, 26 Aug 2026 12:17:40 GMT</pubDate>
            <atom:updated>2026-08-26T12:17:40.679Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article 6 of 10 in the series: Feature Engineering for Time Series Forecasting</em></p><blockquote><strong>Lag features are the single most consequential decision in tree-based time series ML.</strong> Pick the wrong lags and your model has no memory. Pick too many and you burn training signal on noise. Pick the wrong horizon-lag relationship and you leak the future into training without noticing.</blockquote><h3>Why This Article Sits Where It Does</h3><p>Articles 1 through 5 built the infrastructure — tabularization, leakage-safe pipelines, decomposition, imputation, anomaly detection. Everything has been about <em>preparing</em> the series and <em>diagnosing</em> it.</p><p>Article 6 pivots back to the discipline this series is named after: <strong>actually engineering features that let tree-based models compete with (and often beat) classical statistical forecasting models.</strong></p><p>Lag features are the entry point. They’re deceptively simple — you shift the target back in time and use those shifted values as inputs. But which lags to shift by, how many to include, and how to select them are decisions that decide whether your model has any predictive power at all.</p><p>This article covers:</p><ul><li>The three categories of lag features and when each matters</li><li><strong>ACF and PACF</strong> — the classical selection tools, translated for ML practitioners</li><li><strong>Sequential vs seasonal lags</strong> — the pattern that separates working models from broken ones</li><li><strong>Cross-correlation function (CCF)</strong> for exogenous variables</li><li>The horizon-lag rule that prevents an entire class of leakage bugs</li></ul><p>Every technique is anchored in a runnable example. Every visualization comes from the actual code in the article.</p><h3>What a Lag Feature Actually Is</h3><p>A lag feature is the target series shifted back in time by k periods, then included as a column in your feature matrix.</p><pre>df[&#39;target_lag_1&#39;] = series.shift(1)   # yesterday&#39;s value<br>df[&#39;target_lag_7&#39;] = series.shift(7)   # one week ago</pre><p>Recall the tabularization we established in Article 1: your model needs a 2D matrix where each row is a forecasting instance. Lag features are the primary way sequential history becomes columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8Xvze_i2Wuz7Rrz_0bcCg.png" /><figcaption>Figure 1 — Tabularization via lag features. Each row of the tabular view represents one forecasting instance. Columns are past values. The first N rows drop out as NaN because you need enough history to compute the largest lag.</figcaption></figure><p>The trade-off is immediate: <strong>every additional lag you include costs you a row of training data at the start of the series</strong> (because that lag can’t be computed for the earliest observations). For a series with 2 years of daily data, including lag_365 costs you an entire year of training data. This becomes non-trivial for short series or expensive when your longest lag is a significant fraction of your total history.</p><h3>The Three Categories of Lag Features</h3><p>Not every lag serves the same purpose. In production forecasting, lag features fall into three categories.</p><p><strong>Sequential lags</strong> — lag_1, lag_2, lag_3. Capture short-term momentum and autocorrelation. Answer the question <em>&quot;where is the series pointing right now?&quot;</em></p><p><strong>Seasonal lags</strong> — lag_7, lag_14, lag_28 for daily data with weekly seasonality; lag_12, lag_24 for monthly data with yearly seasonality. Capture the phase of the seasonal cycle. Answer <em>&quot;what did this same period look like last cycle?&quot;</em></p><p><strong>Long-horizon lags</strong> — lag_365 for daily data with yearly patterns, lag_52 for weekly data. Capture year-over-year effects, cycle-over-cycle drift.</p><p>Different problems weight these categories differently. Financial time series often care mostly about sequential lags (momentum). Retail demand cares heavily about seasonal lags (same weekday last week). Slow-moving inventory forecasting might care almost entirely about long-horizon lags (this month last year).</p><p><strong>A common mistake:</strong> treating lag selection as a single decision. The right approach is to select from each category deliberately, based on what the data actually shows — which is what the next section is for.</p><h3>Selecting Lags with ACF and PACF</h3><p>The classical tools for lag selection are the <strong>Autocorrelation Function (ACF)</strong> and the <strong>Partial Autocorrelation Function (PACF)</strong>. They come from ARIMA-era statistical forecasting, but they translate cleanly to ML feature selection.</p><p><strong>ACF</strong> measures the correlation between the series and itself at each lag. If y(t) is highly correlated with y(t-7) across the whole series, ACF at lag 7 will spike.</p><p><strong>PACF</strong> measures the <em>direct</em> correlation between the series and its lag at k, controlling for the correlation at all shorter lags. This isolates which lags contribute <em>unique</em> information versus which are just downstream effects of earlier lags.</p><pre>from statsmodels.graphics.tsaplots import plot_acf, plot_pacf<br>import matplotlib.pyplot as plt<br><br>fig, axes = plt.subplots(1, 2, figsize=(15, 5))<br>plot_acf(sales, lags=40, ax=axes[0])<br>plot_pacf(sales, lags=40, ax=axes[1], method=&#39;ywm&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n5cWf2eqn4ZneCNXp_W8Bg.png" /><figcaption>Figure 2 — ACF and PACF on a series with weekly seasonality. ACF shows the characteristic seasonal pattern: spikes at lags 7, 14, 21, 28 that decay slowly. PACF shows what’s unique at each lag — the first few sequential lags carry significant partial correlation, then the signal dies out. This is the diagnostic signature of a series with strong weekly seasonality plus some short-term autocorrelation.</figcaption></figure><p><strong>How to read them for feature selection:</strong></p><ul><li><strong>ACF with periodic spikes</strong> → strong seasonal structure. Include seasonal lags at the peak positions.</li><li><strong>PACF cutting off after a few lags</strong> → those first few lags carry most of the sequential information. Include lags 1 through the cutoff.</li><li><strong>ACF slowly decaying</strong> → non-stationary series (trend not removed). Differencing may help before lag analysis.</li><li><strong>Neither showing clear structure</strong> → the series may be dominated by noise, and lag features alone won’t provide much signal.</li></ul><p>The rule I follow in practice: <strong>include sequential lags 1 through 3 by default, then add seasonal lags at the peaks visible in ACF up to the point where they fall inside the confidence band.</strong> This gives you a defensible feature set without over-including noisy lags.</p><h3>Lag Plots — The Visual Sanity Check</h3><p>ACF and PACF give you correlation numbers. <strong>Lag plots</strong> give you the shape of the relationship. Every practitioner should run these before committing to a lag set.</p><pre>import matplotlib.pyplot as plt<br><br>fig, axes = plt.subplots(2, 3, figsize=(15, 9))<br>for ax, lag in zip(axes.flat, [1, 2, 7, 14, 28, 90]):<br>    lagged = sales.shift(lag)<br>    ax.scatter(lagged, sales, s=8, alpha=0.4)<br>    ax.set_xlabel(f&#39;y(t - {lag})&#39;)<br>    ax.set_ylabel(&#39;y(t)&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5svVuUECuJ9cX844PAOzA.png" /><figcaption>Figure 3 — Lag plots for the same series at 6 different lag values. Strong diagonal alignment (lag 7, 14) indicates high correlation — the model has learned something useful. Diffuse cloud (lag 3, 90) means low correlation. Note how lag 7 shows the tightest cluster around the y=x line because it aligns with the weekly seasonal period.</figcaption></figure><p>Two things to look for:</p><ul><li><strong>Tight diagonal</strong> → strong linear relationship. Lag is likely useful.</li><li><strong>Diffuse cloud</strong> → weak or no relationship. Lag adds noise not signal.</li></ul><p>A less-obvious use: <strong>lag plots reveal non-linearity</strong>. If the scatter shows a clear curved shape rather than a linear one, tree-based models can exploit that non-linearity via splits — but linear regression cannot. This is one of the reasons LightGBM/XGBoost often outperform linear models on time series with complex seasonal-trend interactions.</p><h3>Sequential vs Seasonal Lag Importance</h3><p>Here’s the empirical finding that most practitioners internalize slowly: <strong>for tree-based models on strongly seasonal series, seasonal lags almost always outrank sequential lags in feature importance.</strong></p><p>Let’s demonstrate this. Fit a LightGBM model with both sequential lags (1–6) and seasonal lags (multiples of 7) on the same series, then look at what the model actually values.</p><pre>import lightgbm as lgb<br><br>seq_lags = [1, 2, 3, 4, 5, 6]<br>seasonal_lags = [7, 14, 21, 28, 35, 42, 49, 56]<br>all_lags = seq_lags + seasonal_lags<br><br>df = pd.DataFrame({&#39;target&#39;: sales})<br>for lag in all_lags:<br>    df[f&#39;lag_{lag}&#39;] = sales.shift(lag)<br>df = df.dropna()<br><br>X = df.drop(columns=&#39;target&#39;)<br>y = df[&#39;target&#39;]<br><br>model = lgb.LGBMRegressor(n_estimators=200, learning_rate=0.05,<br>                           max_depth=5, verbose=-1, random_state=42)<br>model.fit(X, y)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NFjpRzE6rbNCgf1JuAWVgQ.png" /><figcaption>Figure 4 — LightGBM feature importances for sequential (gray) and seasonal (red) lags on a weekly-seasonal series. The most important single feature is lag_7 (the exact weekly period). But note that lag_6 also ranks near the top — the model is likely using it as a proxy for “yesterday’s day-of-week” via feature interactions with lag_7. This is a real pattern: sequential and seasonal lags complement each other rather than substituting.</figcaption></figure><p>Three observations worth internalizing.</p><p><strong>Lag 7 dominates.</strong> For a series with weekly seasonality, the exact-cycle lag is the most direct signal available to the model. This is why classical seasonal ARIMA models with S=7 outperform non-seasonal ARIMA — same principle, different model family.</p><p><strong>Sequential lags matter for interaction effects.</strong> LightGBM values lag_6 highly, but not because lag_6 is intrinsically informative — it’s because the interaction between lag_7 and lag_6 lets the model reason about “one day before the same day last week.” Tree splits exploit these interactions in ways linear models cannot.</p><p><strong>More seasonal lags is not always better.</strong> Note the drop between lag_28 and lag_35. Beyond a few weeks of history, additional seasonal lags contribute diminishing information. The right rule is <em>“include seasonal lags until their ACF value falls below the confidence band”</em> — usually 4–8 seasonal lags for weekly-seasonal daily data.</p><h3>Cross-Correlation for Exogenous Variables</h3><p>Everything above assumes you’re selecting lags of the <em>target series itself</em>. But most real forecasting problems include exogenous variables — marketing spend, weather, competitor pricing, foot traffic.</p><p>For exogenous variables, the question is different: <strong>at what lag does this variable predict the target most strongly?</strong></p><p>The tool is the <strong>Cross-Correlation Function (CCF)</strong>.</p><pre>def cross_correlation(target, exog, max_lag=20):<br>    &quot;&quot;&quot;Compute cross-correlation between target and exog at various lags.&quot;&quot;&quot;<br>    ccf_vals = []<br>    lags = range(-max_lag, max_lag + 1)<br>    for lag in lags:<br>        if lag &lt; 0:<br>            corr = target.corr(exog.shift(-lag))<br>        else:<br>            corr = target.corr(exog.shift(lag))<br>        ccf_vals.append(corr)<br>    return list(lags), ccf_vals<br><br>lags, ccf = cross_correlation(sales_with_marketing, marketing, max_lag=20)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UL5krSa421yR0jTm6A-5Aw.png" /><figcaption>Figure 5 — Cross-correlation function between marketing spend and sales. Peak at lag +3 (green) means marketing leads sales by 3 days — spend today drives sales three days later. The green region shows lags where the exogenous variable leads the target (usable as features). The red region shows lags where sales leads marketing (not usable — those would be leakage).</figcaption></figure><p><strong>How to read a CCF for feature engineering:</strong></p><ul><li><strong>Peak at positive lag</strong> → exogenous variable <em>leads</em> the target. Include the variable at that lag as a feature.</li><li><strong>Peak at zero lag</strong> → simultaneous relationship. Usable if the variable is known at prediction time.</li><li><strong>Peak at negative lag</strong> → exogenous variable <em>follows</em> the target. Not usable as a predictive feature (would be leakage).</li></ul><p>The critical rule for production: <strong>only include exogenous features at lags where the variable will actually be observable at prediction time.</strong> If you’re forecasting sales 7 days ahead and marketing spend peaks at CCF lag +3, that means today’s marketing predicts sales 3 days from now — but you’d need marketing data from 4 days ago to have a valid feature for a 7-day forecast. Alignment matters more than correlation strength.</p><h3>The Horizon-Lag Rule</h3><p>The single most common leakage bug in multi-step forecasting comes from misaligning lags with the forecast horizon.</p><p><strong>The rule:</strong> for a forecast horizon of H periods, every lag feature must satisfy lag_k &gt;= H.</p><p>Why? If you’re forecasting y(t+H) from features computed at time t, then a feature like lag_1 would be y(t-1) — which was observed <strong>H+1 periods before the target</strong>. That&#39;s fine. But what about lag_0? That would be y(t) itself, which by definition you don&#39;t know when predicting y(t+H) in production. And lag_-3? Impossible.</p><p>The trap is subtler for multi-step direct forecasting:</p><pre># Building a 12-step-ahead forecast<br>horizon = 12<br>y_target = sales.shift(-horizon)  # target is 12 periods in the future<br><br># ❌ BUG: including lags smaller than horizon<br>X[&#39;lag_1&#39;] = sales.shift(1)   # At forecast time, this is y(t-1)<br>                              # But we&#39;re predicting y(t+12)<br>                              # In production at t, y(t-1) is known: OK<br><br># ✅ Actually, this is subtler than the naive interpretation.</pre><p>Let me be more precise. The bug isn’t that lag_1 is unusable — at prediction time t, y(t-1) is known. The bug is if you&#39;re computing the feature matrix at t but treating it as valid at t+H. If your feature engineering pipeline uses lag_1 and you&#39;re evaluating multi-step forecasts by re-using the same feature matrix, you either need to:</p><ol><li><strong>Recompute features fresh at each forecast origin</strong> (direct multi-step approach, correct)</li><li><strong>Only use lags </strong><strong>&gt;= H</strong> (single-model approach, safer)</li></ol><p>The second option is what most production pipelines default to for a reason — it’s simpler to reason about and it prevents the class of bugs where “recompute features” is silently forgotten in the evaluation loop.</p><p>For our reusable feature builder from Article 1’s companion code:</p><pre>def add_lag_features_horizon_aware(<br>    df, target_col, lags, forecast_horizon=1<br>):<br>    &quot;&quot;&quot;<br>    Add lag features, dropping any lag smaller than the forecast horizon.<br>    &quot;&quot;&quot;<br>    valid_lags = [lag for lag in lags if lag &gt;= forecast_horizon]<br>    if len(valid_lags) &lt; len(lags):<br>        dropped = [lag for lag in lags if lag &lt; forecast_horizon]<br>        print(f&quot;[warn] Dropped lags {dropped} — smaller than forecast horizon.&quot;)<br><br>    result = df.copy()<br>    for lag in valid_lags:<br>        result[f&#39;{target_col}_lag_{lag}&#39;] = df[target_col].shift(lag)<br>    return result</pre><p>This is the pattern encoded in the <a href="https://github.com/Ameen2488/time-series-feature-engineering/blob/main/src/features.py">companion repo’s </a><a href="https://github.com/Ameen2488/time-series-feature-engineering/blob/main/src/features.py">features.py</a>. Every lag operation respects forecast_horizon. Skip this and you&#39;ll ship a model that scores well in CV and fails in production for reasons your test suite won&#39;t catch.</p><p>Decision Framework</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2tKo9t3JYu5cqYY0NRmHtQ.png" /><figcaption>Figure 6 — Decision framework for lag selection. Start from what you know about seasonality, use ACF/PACF to validate, then filter by forecast horizon.</figcaption></figure><p>The framework in text form:</p><ol><li><strong>Do you know the dominant seasonal period?</strong> If yes, include lags at multiples of that period. If no, run ACF/PACF and identify significant spikes.</li><li><strong>Add sequential lags 1–3</strong> for short-term memory.</li><li><strong>Filter by forecast horizon</strong> — drop any lag smaller than your forecast horizon.</li><li><strong>For exogenous variables</strong>, use CCF to find the peak positive lag, then confirm that lag is observable at prediction time.</li></ol><p>Following this framework produces a defensible, minimal feature set — usually 6–12 lag features total. That’s enough for LightGBM to build a strong model without over-fitting to noise.</p><h3>Key Takeaways</h3><ul><li>Lag features are the primary way tree-based models learn from sequential time series data</li><li>Three categories matter: <strong>sequential</strong> (short-term momentum), <strong>seasonal</strong> (same-period-last-cycle), and <strong>long-horizon</strong> (year-over-year)</li><li><strong>ACF and PACF</strong> are the classical selection tools and translate cleanly to ML feature selection — ACF reveals seasonality patterns, PACF isolates unique lag contributions</li><li><strong>Lag plots</strong> are the visual sanity check — tight diagonal means useful, diffuse cloud means noise</li><li>For strongly seasonal series, <strong>seasonal lags typically outrank sequential lags</strong> in tree-model feature importance — but sequential lags are still valuable for the interaction effects trees exploit</li><li>The <strong>Cross-Correlation Function (CCF)</strong> is the equivalent tool for finding predictive lags of exogenous variables</li><li>The <strong>horizon-lag rule</strong> — always drop lags smaller than your forecast horizon — prevents an entire class of leakage bugs</li></ul><h3>What’s Next</h3><p>Article 7 — <em>“Window Features: Rolling, Expanding &amp; Exponential Smoothing”</em> — extends lag features into their statistical-summary counterparts. Rolling means, expanding statistics, exponentially-weighted moving averages, and the specific patterns that make each one appropriate for different forecasting problems. Includes the subtle leakage bugs that specifically affect window features.</p><p><strong>→ Subscribe to the newsletter to get Article 7 next Tuesday.</strong></p><p><em>If you’ve built a lag-based forecasting model in production, drop a comment on the lag set you converged on and why. The seasonal-lag-count decisions especially — where did you land, and what made you cut off there?</em></p><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></blockquote><p>#TimeSeries #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #LightGBM #ACF #PACF</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=005a0493ed32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/lag-features-teaching-your-model-to-remember-the-past-005a0493ed32">Lag Features: Teaching Your Model to Remember the Past</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Outlier Detection in Time Series: Rolling Stats, LOWESS & STL]]></title>
            <link>https://medium.com/data-science-collective/outlier-detection-in-time-series-rolling-stats-lowess-stl-bc2af56a6276?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc2af56a6276</guid>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-08-18T03:31:01.225Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article 5 of 10 in the series: Feature Engineering for Time Series Forecasting</em></p><blockquote><strong>Every anomaly detector is a subtraction.</strong> You subtract what the series “should” look like from what it actually is, and flag the residuals that exceed a threshold. The four methods in this article differ only in how they estimate “should.” That single design choice decides whether your detector catches real anomalies or drowns in false positives.</blockquote><h3>Why This Article Sits Where It Does</h3><p>We’ve spent four articles building the infrastructure. Article 3 gave us decomposition. Article 4 gave us imputation. Both articles ended with a bridge to anomaly detection — because the same tools that clean and structure your series also expose what doesn’t belong in it.</p><p>This article makes that bridge explicit.</p><p>Anomaly detection in time series is not a separate discipline from forecasting. It’s the same problem viewed from the other side. Forecasting asks <em>“given the past, what should the next value be?”</em> Anomaly detection asks <em>“given what should have happened, which observations don’t fit?”</em> Both questions require the same underlying model of the series. The difference is what you do with the residuals.</p><p>Four methods, roughly increasing in sophistication:</p><ol><li><strong>Rolling z-score</strong> — the workhorse of production monitoring</li><li><strong>Rolling IQR bands</strong> — robust to fat-tailed residuals</li><li><strong>LOWESS residuals</strong> — for series with smooth trend but weak seasonality</li><li><strong>STL residuals</strong> — the modern default for series with clear structure</li></ol><p>Each has a right context. This article walks through all four on the same synthetic series, then compares what each one catches and misses.</p><h3>The Three Categories of Anomalies</h3><p>Before choosing a method, you need to know what you’re looking for. Anomalies in time series fall into three categories, each with different detection requirements.</p><p><strong>Point anomalies</strong> — single observations that deviate sharply from their neighbours. A sensor reading of 200 when the surrounding values sit around 100. The simplest and most commonly-detected type.</p><p><strong>Contextual anomalies</strong> — values that look normal in isolation but are wrong given their context. Sales of 500 on a Sunday when Sundays typically average 300. Requires seasonal awareness.</p><p><strong>Collective anomalies</strong> — sequences of observations that are unusual as a run, even if no individual point stands out. Ten consecutive days of slightly-elevated readings signaling a structural shift. Requires pattern recognition beyond single points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOqTcXKs9feOT_xgjye1zg.png" /><figcaption>Figure 1 — The same series with three anomaly types injected. Red dots are point anomalies (single-day spikes and drops). Orange band is a contextual anomaly (a 10-day period where the series shifted upward). Purple band is a collective anomaly (a run of unusually elevated values).</figcaption></figure><p><strong>Why this matters:</strong> rolling z-score catches point anomalies well but often misses contextual and collective ones. STL residuals catch all three because they subtract the seasonal expectation before looking for deviations. Your choice of method has to match the type of anomaly your business cares about.</p><p>If you’re monitoring for equipment failure (point anomalies dominate), simpler methods work. If you’re monitoring demand for structural shifts (contextual and collective anomalies dominate), STL is nearly always the right call.</p><h3>Method 1 — Rolling Z-Score</h3><p>The most-used anomaly detector in production because it’s simple, fast, and works well enough for most operational monitoring.</p><p><strong>The idea:</strong> compute a rolling mean and rolling standard deviation over recent history. Any observation more than k standard deviations away from the rolling mean is flagged.</p><pre>import pandas as pd<br><br>def rolling_zscore(series: pd.Series, window: int = 28) -&gt; pd.Series:<br>    &quot;&quot;&quot;<br>    Compute the rolling z-score of a series. Uses shift(1) before<br>    rolling to prevent the current observation from contaminating<br>    its own baseline.<br>    &quot;&quot;&quot;<br>    rolling_mean = series.shift(1).rolling(window).mean()<br>    rolling_std  = series.shift(1).rolling(window).std()<br>    return (series - rolling_mean) / rolling_std<br><br># Detect anomalies at |z| &gt; 3<br>z_scores = rolling_zscore(sales, window=28)<br>anomalies = z_scores[z_scores.abs() &gt; 3]</pre><p>Two design decisions matter.</p><p>The shift(1) is not optional. Without it, the rolling mean at time t includes y(t) itself — the very observation whose deviation you&#39;re trying to measure. Same leakage rule as Article 1: any statistic used to evaluate y(t) must be computed from values before t.</p><p>The window length trades responsiveness against stability. A 7-day window reacts quickly but generates many false positives when the underlying level shifts. A 90-day window is stable but slow to recognize genuine regime changes as normal. For daily operational data, 28–30 days is a reasonable default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MdEYM33-2hzaKkKKladpYQ.png" /><figcaption>Figure 2 — Rolling z-score detection on the reference series. Top: sales with detected anomalies in red. Bottom: z-scores with ±3σ threshold bands. Most extreme point anomalies get caught, but the method struggles with anomalies that occur near the seasonal trough — the local variance in that region is large enough that even a 45-unit spike doesn’t cross 3σ.</figcaption></figure><p><strong>When rolling z-score fits:</strong> operational metrics that are roughly stationary within their rolling window. Server response times, transaction volumes, sensor readings from stable equipment.</p><p><strong>When it fails:</strong> series with strong seasonality (weekend behavior looks anomalous every week), series with heavy-tailed noise distributions (many false positives), or series undergoing structural change (the rolling mean chases the shift rather than flagging it).</p><h3>Method 2 — Rolling IQR Bands</h3><p>Rolling z-score assumes Gaussian residuals. When residuals are heavy-tailed (a common reality in operational data), the mean and standard deviation get pulled around by extreme values, degrading detection.</p><p>Rolling IQR replaces both statistics with quantiles.</p><pre>def rolling_iqr_bands(<br>    series: pd.Series,<br>    window: int = 28,<br>    k: float = 1.5,<br>) -&gt; tuple[pd.Series, pd.Series]:<br>    &quot;&quot;&quot;<br>    Compute rolling upper/lower anomaly bands using the IQR rule:<br>    lower = Q1 - k * IQR<br>    upper = Q3 + k * IQR<br>    &quot;&quot;&quot;<br>    q1 = series.shift(1).rolling(window).quantile(0.25)<br>    q3 = series.shift(1).rolling(window).quantile(0.75)<br>    iqr = q3 - q1<br>    upper = q3 + k * iqr<br>    lower = q1 - k * iqr<br>    return lower, upper<br><br>lower, upper = rolling_iqr_bands(sales, window=28, k=1.5)<br>anomalies = sales[(sales &gt; upper) | (sales &lt; lower)]</pre><p>The k=1.5 multiplier is the classical Tukey convention (matching boxplot whiskers). Larger k means fewer, more conservative detections; smaller k catches more but risks false positives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8zBTJqnRlGGzjSIMqsXKA.png" /><figcaption>Figure 3 — Rolling IQR bands (green shaded region) with detected anomalies (red). Note how the band adapts to the level and volatility of the series over time. The band width tells you how uncertain “normal” is at each timestamp.</figcaption></figure><p><strong>When rolling IQR fits:</strong> any series where you’d previously reach for rolling z-score but suspect residuals aren’t well-behaved Gaussians. Financial data, click-through rates, any metric with occasional legitimate spikes that would inflate a rolling standard deviation.</p><p><strong>When it fails:</strong> same seasonal-blindness as rolling z-score. Both are essentially local-mean detectors dressed up differently.</p><h3>Method 3 — LOWESS Residual Detection</h3><p>The two rolling methods above have a fundamental limitation: they treat “normal” as <em>stable within a window</em>. That assumption breaks whenever the underlying series has a trend that changes speed, or whenever seasonality creates predictable variance.</p><p>LOWESS (locally-weighted scatterplot smoothing, Article 3) fits a flexible curve to the series and then flags observations whose residuals exceed a threshold.</p><pre>import numpy as np<br>from statsmodels.nonparametric.smoothers_lowess import lowess<br><br>def lowess_anomalies(<br>    series: pd.Series,<br>    frac: float = 0.05,<br>    threshold_sigma: float = 3.0,<br>) -&gt; pd.Series:<br>    &quot;&quot;&quot;<br>    Fit LOWESS to the series. Flag observations whose residuals<br>    exceed threshold_sigma standard deviations from the mean residual.<br>    &quot;&quot;&quot;<br>    smoothed = lowess(<br>        series.values,<br>        np.arange(len(series)),<br>        frac=frac,<br>        return_sorted=False,<br>    )<br>    fit = pd.Series(smoothed, index=series.index)<br>    residuals = series - fit<br><br>    upper = residuals.mean() + threshold_sigma * residuals.std()<br>    lower = residuals.mean() - threshold_sigma * residuals.std()<br>    return residuals[(residuals &gt; upper) | (residuals &lt; lower)]</pre><p>The frac parameter controls how much history each local fit uses. Small frac (0.02-0.05) tracks the series closely — good for spotting spikes but poor at handling smooth trends. Large frac (0.2-0.3) gives a smooth backbone that lets more anomalies stand out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IjQWbmAeIN2BRvBm6Ks3sA.png" /><figcaption>Figure 4 — LOWESS fit (blue) tracks the series through weekly oscillations. Residuals in the bottom panel show clean spikes at anomaly locations, with the ±3σ threshold flagging them. Notice how LOWESS handles the trend cleanly but treats the collective anomaly region as noise rather than a structural event.</figcaption></figure><p><strong>Two production caveats:</strong></p><ul><li>LOWESS fits the entire series at once. In a walk-forward CV setting, you must refit LOWESS on the training window only — same leakage discipline as everywhere else in this series.</li><li>LOWESS is computationally expensive for long series. For real-time detection on high-frequency data, use it to establish baseline thresholds offline, then apply the thresholds online.</li></ul><p><strong>When LOWESS fits:</strong> series with smooth non-linear trends but weak or absent seasonality. Cumulative user counts, gradually-warming sensor readings, click-through rates over a growing user base.</p><h3>Method 4 — STL Residual Detection</h3><p>STL residuals are the modern default for anomaly detection in structured time series. They handle trend, they handle seasonality, they’re robust to outliers when you set robust=True, and they compose cleanly with everything else in this series.</p><pre>from statsmodels.tsa.seasonal import STL<br><br>def stl_anomalies(<br>    series: pd.Series,<br>    period: int = 7,<br>    threshold_sigma: float = 3.0,<br>) -&gt; pd.Series:<br>    &quot;&quot;&quot;<br>    STL residual-based anomaly detection.<br>    Robust=True downweights outliers when fitting.<br>    &quot;&quot;&quot;<br>    stl = STL(series, period=period, robust=True).fit()<br>    residuals = stl.resid<br><br>    upper = residuals.mean() + threshold_sigma * residuals.std()<br>    lower = residuals.mean() - threshold_sigma * residuals.std()<br>    return residuals[(residuals &gt; upper) | (residuals &lt; lower)]</pre><p>We introduced this pattern at the end of Article 3. Here’s what it looks like when applied to a series with all three anomaly types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pJ1B8hlH44wIBH2bVhorFQ.png" /><figcaption>Figure 5 — STL residual-based detection catches most of what the earlier methods missed. Because STL subtracts both trend and weekly seasonality before flagging deviations, the residuals contain only the structural anomalies — including the collective run around November 2024 that Rolling Z-Score and LOWESS both under-detected.</figcaption></figure><p><strong>When STL fits:</strong> any series with clear trend and/or seasonal structure — which describes most business time series. Retail demand, energy consumption, web traffic, industrial monitoring.</p><p><strong>When STL doesn’t fit:</strong> very short series (need at least 2 full seasonal periods), series with no clear seasonality (LOWESS is simpler and equally effective), or series where the “anomalies” are actually regime changes you want to model rather than flag (STL will smooth over gradual shifts).</p><h3>Comparing the Four Methods on the Same Data</h3><p>Nothing sharpens the differences like putting all four on one plot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XBfAF3l9OfxSA4GAF1k6BQ.png" /><figcaption>Figure 6 — Four detectors on the same series. Black stars mark the six true point anomalies I injected. Each detector’s flags appear as coloured tick marks above the series. Rolling Z-Score and IQR each catch 5 of 6 point anomalies but miss the ones near the seasonal trough. LOWESS catches 4. STL catches all 6 plus flags the collective anomaly cluster around Nov 2024 that the others don’t touch.</figcaption></figure><p>Two observations worth internalizing.</p><p><strong>First — count of detections is not the same as quality of detections.</strong> STL flags 10 anomalies to Rolling Z-Score’s 5, but that difference includes 4 legitimate collective-anomaly detections plus a small handful of false positives. Whether that trade-off is right for your use case depends on the cost of false positives versus missed anomalies in your specific application.</p><p><strong>Second — the methods are complementary, not competitive.</strong> In production, running two detectors and requiring both to agree (an “AND” rule) is a robust way to reduce false positives. Running two detectors and flagging when either agrees (an “OR” rule) increases recall at the cost of precision. Which rule you pick depends on whether the downstream response to an anomaly is expensive (page a human) or cheap (log for later review).</p><h3>The Production Pattern: Detector as a Feature</h3><p>Everything above treats anomaly detection as an end in itself. But in a forecasting pipeline, the more useful pattern is:</p><p><strong>Use the detector to create a feature that the forecasting model consumes.</strong></p><pre>def add_anomaly_features(<br>    series: pd.Series,<br>    period: int = 7,<br>) -&gt; pd.DataFrame:<br>    &quot;&quot;&quot;<br>    Build anomaly-related features from a time series:<br>    - The STL residual value itself<br>    - Rolling std of STL residuals (regime-shift signal)<br>    - Binary flag for extreme residuals<br>    &quot;&quot;&quot;<br>    stl = STL(series, period=period, robust=True).fit()<br>    residuals = stl.resid<br><br>    threshold = 3 * residuals.std()<br><br>    features = pd.DataFrame(index=series.index)<br>    features[&#39;stl_residual&#39;] = residuals<br>    features[&#39;residual_volatility&#39;] = residuals.shift(1).rolling(14).std()<br>    features[&#39;is_extreme_residual&#39;] = (residuals.abs() &gt; threshold).astype(int)<br>    features[&#39;days_since_last_anomaly&#39;] = (<br>        (~features[&#39;is_extreme_residual&#39;].astype(bool))<br>        .groupby(features[&#39;is_extreme_residual&#39;].cumsum())<br>        .cumcount()<br>    )<br><br>    return features</pre><p>This gives your forecasting model direct access to “this observation is unusual” information, so it can learn to weight recent anomalies differently — often improving forecast robustness after volatile periods.</p><p>The same critical warning applies as everywhere in this series: <strong>the STL fit must be computed inside your walk-forward CV loop using only training data.</strong> Fitting STL on the full series, extracting anomaly features, then using those features for training leaks the future into your model. This is the subtle bug that undoes months of careful feature engineering if you’re not paying attention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mRbNvKoVYBpCUqSZm1ZEfQ.png" /></figure><p>For a general starting point on unknown data: <strong>STL residuals with </strong><strong>robust=True, </strong><strong>period set to your dominant seasonal cycle, and a 3σ threshold.</strong> Same conclusion as Article 3 — STL earns its keep across more edge cases than any other single method.</p><h3>Key Takeaways</h3><ul><li>Every anomaly detector is a subtraction — subtracting what “should” happen from what did, and flagging the residuals</li><li>The three anomaly categories (<strong>point / contextual / collective</strong>) require different detection strategies; know which you’re looking for before choosing a method</li><li><strong>Rolling z-score</strong> and <strong>rolling IQR</strong> are the operational workhorses; simple, fast, but seasonally blind</li><li><strong>LOWESS residuals</strong> handle smooth trends but poorly capture strong seasonality</li><li><strong>STL residuals</strong> are the modern default for series with clear structure — handle trend, seasonality, and are outlier-robust</li><li>The more useful production pattern is <strong>anomaly-derived features feeding a forecasting model</strong>, not standalone detection</li><li>Every detector must be <strong>refitted inside walk-forward CV</strong> to stay leakage-safe</li></ul><h3>What’s Next</h3><p>Article 6 — <em>“Lag Features: Teaching Your Model to Remember the Past”</em> — pivots from cleaning and monitoring back to the core feature engineering discipline. Which lags to include, how to select them using ACF and PACF, why seasonal lags matter more than sequential ones for tree-based models, and the specific patterns that emerge when you visualize lag-target relationships.</p><p><strong>→ Subscribe to the newsletter to get Article 6 next Tuesday.</strong></p><blockquote><em>If you’ve shipped an anomaly detector in production, drop a comment on which method you settled on and why. The false-positive stories are always the best teachers.</em></blockquote><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></blockquote><p>#TimeSeries #AnomalyDetection #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #STL #MLOps</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc2af56a6276" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/outlier-detection-in-time-series-rolling-stats-lowess-stl-bc2af56a6276">Outlier Detection in Time Series: Rolling Stats, LOWESS &amp; STL</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Missing Data in Time Series: Beyond Simple Imputation]]></title>
            <link>https://blog.gopenai.com/missing-data-in-time-series-beyond-simple-imputation-edae6e2543c7?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/edae6e2543c7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[missing-data]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 13:16:53 GMT</pubDate>
            <atom:updated>2026-08-14T11:03:27.298Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article 4 of 10 in the series: Feature Engineering for Time Series Forecasting</em></p><blockquote><strong>Sklearn’s </strong><strong>SimpleImputer(strategy=&#39;mean&#39;) is one of the most confidently wrong choices you can make in time series ML.</strong> It&#39;s what most tutorials teach. It also silently destroys the very structure your model exists to learn. This article covers what to do instead.</blockquote><h3>Why Missing Data in Time Series Is a Different Problem</h3><p>Missing data in tabular ML is a well-solved problem. Fit an imputer on the training data using the column mean, median, or mode. Transform the test data with the same statistics. Done.</p><p>That approach falls apart the moment you apply it to a time series.</p><p>The reason is structural: <strong>a time series’s value at time </strong><strong>t is not independent of its neighbours.</strong> If y(t-1) = 145 and y(t+1) = 148, then y(t) being <em>missing</em> is nowhere near being drawn from the series mean of 120. The missing observation belongs to the local neighbourhood, not the global distribution.</p><p>Standard imputation ignores that. Time-series-aware imputation is built around it.</p><p>This distinction matters because missing data in production time series is universal. Sensor dropouts, system outages, promotional events that skip normal reporting, weekends where SKUs aren’t sold, migrations between data warehouses — every real-world series has gaps. What you do with those gaps affects every feature you engineer downstream in this series (lag features, window statistics, decomposition-based features) and every model you train on the result.</p><h3>The Three Patterns of Missing Data</h3><p>Before choosing an imputation method, you have to know what pattern of missingness you’re dealing with. Rubin’s classical taxonomy from 1976 still applies:</p><p><strong>MCAR — Missing Completely At Random.</strong> The probability of a value being missing is independent of both the observed and unobserved values. Rare in practice, but a useful baseline. Example: an intermittent network glitch that drops readings uniformly.</p><p><strong>MAR — Missing At Random.</strong> The probability of missingness depends on other observed variables but not on the missing value itself. Example: sales data missing more often on weekends because the store closes early. The missingness pattern correlates with a <em>known</em> feature (day of week).</p><p><strong>MNAR — Missing Not At Random.</strong> The probability of missingness depends on the missing value itself, or on unobserved factors. Example: a sensor stops reporting when its reading exceeds a threshold, or a 30-day outage caused by a system migration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bkny5pdXQY4b-3CoFrbcMA.png" /><figcaption>Figure 1 — The same underlying series with three different missingness patterns. MCAR scatters gaps uniformly, MAR concentrates them on specific weekdays, and MNAR creates an extended structured gap that no imputation method fully recovers.</figcaption></figure><p><strong>Why this matters:</strong> the imputation method you choose has to match the pattern.</p><ul><li>MCAR forgives almost any reasonable imputation method</li><li>MAR requires you to condition on the observable variable driving the missingness (weekly seasonality, in the weekend example)</li><li>MNAR is the hardest case, and often the right answer is not to impute at all — it’s to explicitly flag the outage and let the model learn to handle it</li></ul><p>Diagnose before you impute. If your missingness pattern is MNAR (a system outage), the “correct” imputation might be NaN plus a was_missing flag, not a filled value. We&#39;ll cover that at the end.</p><h3>The Simple Methods — What They Do, When They Fail</h3><p>Pandas ships four simple imputation methods that are usually the first thing to try:</p><pre># Forward fill — carry the last observation forward<br>series.ffill()<br><br># Backward fill — carry the next observation backward<br>series.bfill()<br><br># Linear interpolation — draw a straight line between neighbours<br>series.interpolate(method=&#39;linear&#39;)<br><br># Mean fill — the classical WRONG answer for TS<br>series.fillna(series.mean())</pre><p>Here’s how each behaves on a series with 15% MCAR gaps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vAbqGHhXaCh94b5xEAy7vw.png" /><figcaption>Figure 2 — Four simple imputation methods on the same data. Forward and backward fill create small step artifacts. Linear interpolation smooths cleanly. Mean imputation collapses everything to a horizontal line at the global mean — losing all local structure.</figcaption></figure><p><strong>Guidance for each:</strong></p><ul><li><strong>ffill()</strong> — safe when the series changes slowly and gaps are short (1-2 periods). Preserves the value from the last known observation. Common in financial time series where the &quot;last known price&quot; carries information.</li><li><strong>bfill()</strong> — same as ffill but backward. Dangerous in forecasting contexts because it can leak future information. <strong>Never use bfill on training data that will be used to forecast forward</strong> — it puts information from t+1 into the imputed value at t.</li><li><strong>interpolate(method=&#39;linear&#39;)</strong> — the sensible default for short gaps in smooth series. Reasonable choice when the underlying process is well-approximated by a line over the missing window.</li><li><strong>fillna(series.mean())</strong> — the answer sklearn&#39;s SimpleImputer gives you and the wrong answer in almost every time series context. It destroys local structure entirely, replacing it with a global average that has no relationship to the actual value at that time.</li></ul><p><strong>The critical warning about linear interpolation:</strong> it works well only when the gap is small relative to the series’s dominant frequency. Interpolate through a 21-day gap in a weekly-seasonal series and you draw a flat line through six weekly peaks. That’s the failure mode most tutorials don’t warn about.</p><h3>Seasonal-Aware Imputation</h3><p>For any series with clear seasonality, the right answer is almost never linear interpolation. It’s <strong>filling with the same-period value from a previous cycle.</strong></p><p>The intuition: if today is a Wednesday and it’s missing, the best single-point estimate isn’t yesterday’s Tuesday value or tomorrow’s Thursday value. It’s <em>last Wednesday’s</em> value — the same phase of the seasonal cycle.</p><pre>def seasonal_fill(series: pd.Series, period: int = 7) -&gt; pd.Series:<br>    &quot;&quot;&quot;<br>    Fill missing values with the same-period value from previous cycles.<br>    Falls back to linear interpolation for any remaining gaps.<br>    &quot;&quot;&quot;<br>    filled = series.copy()<br>    for i in range(len(filled)):<br>        if pd.isna(filled.iloc[i]) and i &gt;= period:<br>            # Try one period back; if also missing, try two periods back<br>            lookback = period<br>            while lookback &lt;= 4 * period and \<br>                  pd.isna(filled.iloc[i - lookback]) and \<br>                  (i - lookback &gt;= 0):<br>                lookback += period<br>            if i - lookback &gt;= 0 and not pd.isna(filled.iloc[i - lookback]):<br>                filled.iloc[i] = filled.iloc[i - lookback]<br>    # Fill remaining gaps at series start with linear<br>    return filled.interpolate(method=&#39;linear&#39;)<br><br>filled = seasonal_fill(sales_with_gaps, period=7)</pre><p>Compare the three approaches on a 25%-missing series with clear weekly seasonality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXVaNWTkq6mpqmzaxe7cNA.png" /><figcaption>Figure 3 — Same series, same gaps, three different methods. Mean imputation flattens everything. Linear interpolation smooths through weekly peaks. Seasonal fill preserves the actual pattern of the series.</figcaption></figure><p>This is a two-line change to your code with an enormous difference in output quality. Any downstream feature that depends on the shape of the series — rolling statistics, lag features, seasonal decomposition — will be dramatically better on the seasonally-imputed series than on the mean-imputed one.</p><h3>STL-Based Imputation — For Large Gaps</h3><p>For extended structured gaps (system outages, migrations, planned downtime), even seasonal fill starts to break down. If you’re missing 21 consecutive days, you need to reconstruct both the underlying trend <em>and</em> the seasonal component simultaneously.</p><p>The pattern that works in production:</p><ol><li>Apply a rough imputation (linear interpolation) to enable STL to fit</li><li>Fit STL on the roughly-filled series</li><li>Reconstruct missing values as trend(t) + seasonal(t) from the STL fit</li></ol><pre>from statsmodels.tsa.seasonal import STL<br><br>def stl_imputation(series: pd.Series, period: int = 7) -&gt; pd.Series:<br>    &quot;&quot;&quot;<br>    STL-based imputation. Preserves both trend and seasonality<br>    through extended gaps.<br>    &quot;&quot;&quot;<br>    # Step 1: rough interpolation to enable STL to fit<br>    rough_fill = series.interpolate(method=&#39;linear&#39;)<br><br>    # Step 2: STL decomposition on the rough fill<br>    stl = STL(rough_fill, period=period, robust=True).fit()<br><br>    # Step 3: reconstruct missing values from trend + seasonal<br>    reconstructed = stl.trend + stl.seasonal<br><br>    # Step 4: use reconstructed values only where original was missing<br>    filled = series.copy()<br>    missing_mask = series.isna()<br>    filled[missing_mask] = reconstructed[missing_mask]<br><br>    return filled</pre><p>The comparison against linear interpolation on a 21-day gap is dramatic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PBpQTlnwUJC8wkq3tPGRBg.png" /><figcaption>Figure 4 — A 21-day gap on the same series. Linear interpolation draws a flat line through the missing period. STL-based imputation reconstructs both the level and the weekly seasonality. Every downstream feature computed on the missing period will be dramatically more accurate on the bottom panel.</figcaption></figure><p><strong>Two important caveats:</strong></p><ul><li><strong>The STL fit must respect train/test boundaries</strong> in cross-validation. Fitting STL on the entire series (including test data), then using its outputs to impute training data, leaks future information. Recompute the STL fit inside your walk-forward CV loop — same warning as Article 3.</li><li>STL-based imputation cannot invent structure that isn’t already present. If your gap includes a promotional event or an external shock, the imputation will smooth over it, potentially masking real signal. Fill with awareness of what was actually happening in the missing window.</li></ul><p>For truly extended gaps (multiple months in daily data), consider whether imputation is even the right approach. Sometimes the right answer is to treat the pre-gap and post-gap periods as separate series with a regime break in between, and let the model learn from both without pretending you have continuous data.</p><h3>Missingness Is a Feature</h3><p>The single most under-used technique in time series ML: <strong>the fact that a value was missing often carries more signal than the imputed value itself.</strong></p><p>Consider a demand forecasting series where a store’s SKU shows missing sales on specific days. If those missing days correlate with promotions (staff too busy to log data), stockouts (nothing to sell), or system migrations, the <em>pattern of missingness</em> encodes information the imputed value doesn’t.</p><p>The fix is trivial:</p><pre>def add_missingness_features(df: pd.DataFrame, target_col: str) -&gt; pd.DataFrame:<br>    &quot;&quot;&quot;<br>    Add a binary flag capturing where the original series was missing,<br>    before imputation.<br>    &quot;&quot;&quot;<br>    df = df.copy()<br>    df[f&#39;{target_col}_was_missing&#39;] = df[target_col].isna().astype(int)<br>    return df<br><br># Then impute normally<br>df = add_missingness_features(df, &#39;sales&#39;)<br>df[&#39;sales&#39;] = stl_imputation(df[&#39;sales&#39;], period=7)</pre><p>Now your model has two signals: the (imputed) sales value, and a flag indicating whether that value was originally observed. Tree-based models pick up interactions between these features naturally — for example, “when sales are high AND was_missing=1, this is likely a stockout-then-restock pattern.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PEQt2QES1_1KsWzfPmx4Q.png" /><figcaption>Figure 5 — Even after imputation, the pattern of when values were missing carries information. In this simulated example, missing days coincide with promotions — the imputed values don’t reflect the true drop, but the was_missing flag preserves the fact that something was different about those days.</figcaption></figure><p>For extended missingness (long gaps), a single binary flag isn’t enough. Add features that capture the <em>structure</em> of the missingness:</p><pre># Days since last observed value<br>df[&#39;days_since_observed&#39;] = (<br>    df[target_col].notna().cumsum().eq(df[target_col].notna().cumsum().shift())<br>    .cumsum()<br>)<br><br># Length of the current run of missing values<br>df[&#39;run_of_missing&#39;] = (<br>    df[target_col].isna().astype(int)<br>    .groupby(df[target_col].notna().cumsum()).cumsum()<br>)</pre><p>These features let the model reason about the <em>quality</em> of the imputed data — a recently-observed value is more reliable than a value from the middle of a 20-day gap.</p><h3>The Complete Missing-Data Pipeline</h3><p>Combining everything from this article into a production-ready pipeline:</p><pre>import pandas as pd<br>from statsmodels.tsa.seasonal import STL<br><br>def robust_time_series_imputation(<br>    series: pd.Series,<br>    period: int = 7,<br>    gap_threshold: int = 3,<br>) -&gt; pd.DataFrame:<br>    &quot;&quot;&quot;<br>    Production imputation pipeline.<br><br>    - Short gaps (&lt;= gap_threshold): linear interpolation<br>    - Long gaps (&gt; gap_threshold): STL-based reconstruction<br>    - Always adds a was_missing flag<br>    &quot;&quot;&quot;<br>    original_missing = series.isna().copy()<br><br>    # Identify gap sizes<br>    gap_ids = original_missing.astype(int).diff().ne(0).cumsum()<br>    gap_sizes = original_missing.groupby(gap_ids).transform(&#39;sum&#39;)<br><br>    # Split short vs long gaps<br>    short_gap_mask = original_missing &amp; (gap_sizes &lt;= gap_threshold)<br>    long_gap_mask = original_missing &amp; (gap_sizes &gt; gap_threshold)<br><br>    filled = series.copy()<br><br>    # Handle short gaps with linear interpolation<br>    filled = filled.interpolate(method=&#39;linear&#39;)<br><br>    # Overwrite long-gap fills with STL-based imputation<br>    if long_gap_mask.any():<br>        rough = filled.copy()<br>        stl = STL(rough, period=period, robust=True).fit()<br>        reconstructed = stl.trend + stl.seasonal<br>        filled[long_gap_mask] = reconstructed[long_gap_mask]<br><br>    result = pd.DataFrame({<br>        &#39;value&#39;: filled,<br>        &#39;was_missing&#39;: original_missing.astype(int),<br>        &#39;gap_size_at_position&#39;: gap_sizes.fillna(0).astype(int),<br>    })<br>    return result<br><br># Usage<br>imputed_df = robust_time_series_imputation(sales_with_gaps, period=7)</pre><p>This is the pattern I use in production. It handles short gaps efficiently, uses STL for long gaps, and preserves the missingness information as features. The whole thing plugs into a sklearn Pipeline via FunctionTransformer if you want it in an end-to-end trainable workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IjF1W0UaX_3loJLjxmr2xA.png" /></figure><p><strong>Always add the </strong><strong>was_missing flag.</strong> It costs one column and often improves downstream model accuracy noticeably.</p><p><strong>Never use </strong><strong>SimpleImputer(strategy=&#39;mean&#39;) on time series data.</strong> It&#39;s the sklearn default. It&#39;s almost always wrong.</p><h3>Key Takeaways</h3><ul><li>Time series missing data is <strong>structurally different</strong> from tabular missing data — values are not independent, so global statistics like the mean are the wrong reference point</li><li>The three missingness patterns (<strong>MCAR / MAR / MNAR</strong>) each require different treatments; diagnose before imputing</li><li>Simple methods work for short gaps: <strong>ffill</strong> for finance-style series, <strong>linear interpolation</strong> for smooth series, <strong>seasonal fill</strong> for cyclical series</li><li>Long gaps need <strong>STL-based imputation</strong> to reconstruct both trend and seasonality</li><li><strong>Missingness is a feature.</strong> Always add a was_missing flag alongside your imputed values — the pattern of missingness often carries more signal than the imputed value itself</li><li>Every imputation step must <strong>respect train/test boundaries</strong> in cross-validation</li></ul><h3>What’s Next</h3><p>Article 5 — <em>“Outlier Detection in Time Series: Rolling Stats, LOWESS &amp; STL”</em> — bridges directly from decomposition and imputation into the anomaly detection thread. Rolling z-scores, IQR bands, LOWESS residuals, STL residual thresholds, and how each technique compounds with the feature infrastructure we’ve built in Articles 1–4.</p><p><strong>→ Subscribe to the newsletter to get Article 5 next Tuesday.</strong></p><p><em>If you’ve shipped a time series model where imputation choices bit you in production, drop a comment on what pattern surprised you. The MNAR cases especially — the outages that look like nothing at all until you look at your model’s failure modes six months later.</em></p><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering">github.com/Ameen2488/time-series-feature-engineering</a></blockquote><p>#TimeSeries #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #MissingData #DataQuality</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edae6e2543c7" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/missing-data-in-time-series-beyond-simple-imputation-edae6e2543c7">Missing Data in Time Series: Beyond Simple Imputation</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decomposing Time Series: Separating Signal from Noise]]></title>
            <link>https://blog.gopenai.com/decomposing-time-series-separating-signal-from-noise-f76b05f2bc9f?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f76b05f2bc9f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 04 Aug 2026 08:48:02 GMT</pubDate>
            <atom:updated>2026-08-11T10:19:06.520Z</atom:updated>
            <content:encoded><![CDATA[<p>Article 3 of 10 in the series: Feature Engineering for Time Series Forecasting</p><blockquote><strong>Every time series is a sum of things you can explain and things you can’t. Decomposition is how you separate them — and it’s the shared first step for forecasting, anomaly detection, and demand pattern analysis.</strong> Get this wrong and every feature you engineer downstream is built on unstable ground.</blockquote><h3>Why Decomposition Is the First Real Analysis Step</h3><p>In Article 1 we tabularized a time series into a feature matrix. In Article 2 we built leakage-safe pipelines around it. Both articles treated the series as a single stream of values.</p><p>But every real time series is actually a <strong>composite of underlying components</strong>:</p><ul><li><strong>Trend</strong> — the long-term direction of the series</li><li><strong>Seasonality</strong> — patterns that repeat over a fixed period</li><li><strong>Cyclicality</strong> — patterns that repeat over variable, longer periods</li><li><strong>Residual (noise)</strong> — what’s left after the structured components are removed</li></ul><p>If you engineer features without knowing what’s driving your series, you end up doing three things poorly at once: modeling trend with tools designed for stationary data, capturing seasonality with lag features that misalign, and treating structural change as noise your model tries to fit.</p><p>Decomposition is the diagnostic that tells you which of these you’re dealing with. It’s also the foundation for a specific class of features (Article 8 and 9 build on this directly) and one of the two most common approaches to anomaly detection in time series.</p><h3>The Two Decomposition Models</h3><p>Every classical decomposition assumes the series was generated by one of two structures.</p><p><strong>Additive model:</strong></p><pre>y(t) = trend(t) + seasonality(t) + residual(t)</pre><p>Use when the seasonal amplitude is roughly constant over time. Sales that swing by ±20 units every week whether the baseline is 100 or 500. Common in fully-scaled operational data.</p><p><strong>Multiplicative model:</strong></p><pre>y(t) = trend(t) × seasonality(t) × residual(t)</pre><p>Use when seasonal amplitude scales with the level of the series. Sales that swing by ±20% every week — so ±20 units at baseline 100, ±100 units at baseline 500. Common in retail, e-commerce, growth-phase businesses.</p><p><strong>The visual test:</strong> plot the raw series. If the peaks and troughs get wider over time, use multiplicative. If they stay the same amplitude, use additive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BCXAYiR44WS6pd0vm772hA.png" /><figcaption>Figure 1 — The visual test for additive vs multiplicative. Left: seasonal amplitude stays constant. Right: seasonal amplitude scales with the level of the series.</figcaption></figure><p><strong>The practical workaround:</strong> log-transform the series first, then apply additive decomposition. Log converts multiplication into addition, so log(y) = log(trend) + log(seasonality) + log(residual). This is often cleaner than fighting with multiplicative decomposition directly.</p><h3>Classical Decomposition — What Most Tutorials Teach</h3><p>Classical decomposition uses moving averages to estimate trend, then subtracts (or divides) to isolate the seasonal component.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from statsmodels.tsa.seasonal import seasonal_decompose<br><br># Generate a series with trend + weekly seasonality + noise<br>np.random.seed(42)<br>dates = pd.date_range(&#39;2022-01-01&#39;, periods=730, freq=&#39;D&#39;)<br>t = np.arange(730)<br>trend = 100 + 0.05 * t<br>seasonality = 15 * np.sin(2 * np.pi * t / 7)  # weekly cycle<br>noise = np.random.normal(0, 5, 730)<br>sales = pd.Series(trend + seasonality + noise, index=dates, name=&#39;sales&#39;)<br><br># Classical additive decomposition<br>result = seasonal_decompose(sales, model=&#39;additive&#39;, period=7)<br><br>fig, axes = plt.subplots(4, 1, figsize=(14, 10), sharex=True)<br>result.observed.plot(ax=axes[0], title=&#39;Observed&#39;)<br>result.trend.plot(ax=axes[1], title=&#39;Trend&#39;)<br>result.seasonal.plot(ax=axes[2], title=&#39;Seasonality&#39;)<br>result.resid.plot(ax=axes[3], title=&#39;Residual&#39;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6lz6bpDkeSaip9rbVpxBuw.png" /><figcaption>Figure 2 — Classical additive decomposition. The trend is smooth, the weekly seasonality is clean, and the residual looks like noise around zero. This is what “well-decomposed” looks like</figcaption></figure><p>The output separates the series into its four components. In a well-decomposed series:</p><ul><li><strong>Trend</strong> should be smooth and interpretable</li><li><strong>Seasonality</strong> should show a clean repeating pattern</li><li><strong>Residual</strong> should look like white noise — no visible structure</li></ul><p><strong>When the residual has visible structure, your decomposition failed to capture something.</strong> This is the diagnostic — a residual that still looks like a wave means you missed a seasonality; a residual with an obvious slope means the trend estimate is wrong.</p><h3>The Problem With Classical Decomposition</h3><p>Classical decomposition has three real limitations that matter in production:</p><ol><li><strong>Rigid seasonal component.</strong> The seasonal pattern is assumed constant across the whole series. If your weekly pattern shifts (e.g., promotions changed shopping behavior mid-year), classical decomposition averages it away.</li><li><strong>Endpoint gaps.</strong> The moving average leaves NaN values at the start and end of the series equal to half the window size. You lose the most recent observations — which is exactly where forecasting needs the diagnostic.</li><li><strong>No robustness to outliers.</strong> A single spike in the raw series contaminates both the trend and the seasonal estimate at that timestamp.</li></ol><p>For toy examples, classical decomposition is fine. For production, it’s usually not the right tool.</p><h3>STL — Seasonal-Trend Decomposition Using LOESS</h3><p>STL (Cleveland et al., 1990) solves all three of classical decomposition’s problems and has become the default modern method. It works by:</p><ol><li>Fitting a LOESS smoother to estimate the trend</li><li>Subtracting the trend to get a detrended series</li><li>Fitting a separate LOESS smoother to each seasonal sub-cycle (e.g., all Mondays, all Tuesdays)</li><li>Iterating until convergence</li></ol><p>The result is a decomposition where <strong>the seasonal component is allowed to evolve slowly</strong> over the series — Mondays in January can look different from Mondays in July if the data says so.</p><pre>from statsmodels.tsa.seasonal import STL<br><br>stl = STL(sales, period=7, robust=True)<br>result = stl.fit()<br><br>fig, axes = plt.subplots(4, 1, figsize=(14, 10), sharex=True)<br>result.observed.plot(ax=axes[0], title=&#39;Observed&#39;)<br>result.trend.plot(ax=axes[1], title=&#39;STL Trend&#39;)<br>result.seasonal.plot(ax=axes[2], title=&#39;STL Seasonality&#39;)<br>result.resid.plot(ax=axes[3], title=&#39;STL Residual&#39;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GqMGbAd_ihc9aInihm15WA.png" /><figcaption>Figure 3 — STL decomposition on the same series. Compare with Figure 2: the trend has no endpoint gaps, the seasonal component is more precisely estimated, and the residuals contain no visible structural pattern.</figcaption></figure><p>The robust=True argument is important. It downweights outliers when fitting the LOESS smoothers, so a single sales spike from a promo doesn&#39;t distort your trend estimate. In production, always use robust=True unless you have a specific reason not to.</p><h3>Tuning STL Parameters</h3><p>STL has three parameters that control how the decomposition behaves:</p><ul><li>period — the seasonal period (7 for weekly on daily data, 12 for monthly on monthly data, 24 for hourly on daily seasonality)</li><li>seasonal — the smoothing window for the seasonal component (must be odd, ≥7). Higher = smoother seasonal pattern that evolves less over time</li><li>trend — the smoothing window for the trend. Higher = smoother trend that responds less to short-term shifts</li></ul><p>The defaults are usually reasonable. When they’re not, the pattern to reach for is:</p><ul><li><strong>Seasonal component looks noisy</strong> → increase seasonal</li><li><strong>Seasonal component is too rigid</strong> → decrease seasonal</li><li><strong>Trend fits every wiggle</strong> → increase trend</li><li><strong>Trend misses obvious level shifts</strong> → decrease trend</li></ul><h3>Multiple Seasonalities — MSTL</h3><p>Real business data often has more than one seasonal pattern. Retail data has both weekly (weekend spikes) and yearly (holiday) seasonality. Energy consumption has daily, weekly, and yearly cycles. Classical decomposition and standard STL only handle one seasonal period.</p><p><strong>MSTL</strong> (Multiple Seasonal-Trend decomposition using Loess) extends STL to handle multiple seasonalities simultaneously.</p><pre>from statsmodels.tsa.seasonal import MSTL<br><br># Series with BOTH weekly AND yearly seasonality (3 years of daily data)<br>dates_long = pd.date_range(&#39;2022-01-01&#39;, periods=1095, freq=&#39;D&#39;)<br>t_long = np.arange(1095)<br>weekly = 15 * np.sin(2 * np.pi * t_long / 7)<br>yearly = 30 * np.sin(2 * np.pi * t_long / 365.25)<br>trend = 100 + 0.05 * t_long<br>noise = np.random.normal(0, 5, 1095)<br>sales_multi = pd.Series(trend + weekly + yearly + noise, index=dates_long)<br><br>mstl = MSTL(sales_multi, periods=[7, 365])<br>result = mstl.fit()<br><br>fig, axes = plt.subplots(5, 1, figsize=(14, 12), sharex=True)<br>result.observed.plot(ax=axes[0], title=&#39;Observed&#39;)<br>result.trend.plot(ax=axes[1], title=&#39;Trend&#39;)<br>result.seasonal[&#39;seasonal_7&#39;].plot(ax=axes[2], title=&#39;Weekly Seasonality&#39;)<br>result.seasonal[&#39;seasonal_365&#39;].plot(ax=axes[3], title=&#39;Yearly Seasonality&#39;)<br>result.resid.plot(ax=axes[4], title=&#39;Residual&#39;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*47TISkjPwxzWgLLawakCeQ.png" /><figcaption>Figure 4 — MSTL cleanly separates weekly and yearly seasonality from the same series. Note how the two seasonalities operate at completely different frequencies — the weekly component oscillates rapidly, the yearly component sweeps slowly across the whole plot.</figcaption></figure><p>For daily retail data, periods=[7, 365] is the standard specification — weekly and yearly. For hourly data, you&#39;d use periods=[24, 168, 8766] for daily, weekly, and yearly cycles respectively. Note that MSTL requires the series to be at least twice as long as the largest period — you cannot fit yearly seasonality on less than 2 years of data.</p><h3>LOWESS Smoothing — When You Just Want the Trend</h3><p>Sometimes you don’t need a full decomposition. You just need to extract the underlying trend for visualization, feature engineering, or outlier detection. LOWESS (locally-weighted scatterplot smoothing) is the tool.</p><pre>from statsmodels.nonparametric.smoothers_lowess import lowess<br><br># Extract just the trend using LOWESS<br>smoothed = lowess(sales.values, np.arange(len(sales)), frac=0.1, return_sorted=False)<br>trend_series = pd.Series(smoothed, index=sales.index)<br><br>fig, ax = plt.subplots(figsize=(14, 4))<br>sales.plot(ax=ax, label=&#39;Raw&#39;, alpha=0.5)<br>trend_series.plot(ax=ax, label=&#39;LOWESS Trend&#39;, linewidth=2, color=&#39;crimson&#39;)<br>ax.legend()<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4KiHVCWWmmGA38Yyya2iXw.png" /><figcaption>Figure 5 — LOWESS trend cutting through the noisy weekly oscillations. The frac=0.1 parameter controls the smoothing window as a fraction of the series length.</figcaption></figure><p>The frac parameter controls smoothing aggressiveness. Values between 0.05 (very local, follows short-term shifts) and 0.3 (very smooth, captures only long-term direction) are typical for daily data.</p><p>LOWESS is particularly useful for <strong>outlier detection</strong> — points where the raw series deviates far from the LOWESS trend are candidates for anomalies. We’ll build this pattern out in Article 5.</p><h3>Using Decomposition Outputs as Features</h3><p>The whole reason for a feature engineering series to cover decomposition is that decomposition outputs make excellent features. Two patterns.</p><h3>Pattern 1: Feed the trend directly</h3><pre># Compute trend on training data, use it as a feature<br>stl = STL(sales, period=7, robust=True).fit()<br><br>features = pd.DataFrame({<br>    &#39;target&#39;: sales,<br>    &#39;stl_trend&#39;: stl.trend,<br>    &#39;stl_seasonal&#39;: stl.seasonal,<br>})</pre><p><strong>Critical:</strong> to keep this leakage-safe in production, you must recompute the STL fit inside your walk-forward cross-validation loop, using only training data. Fitting STL on the full series (including test) and then using its outputs as features leaks future information into training. This is a subtle but common bug.</p><h3>Pattern 2: Use residuals for anomaly features</h3><pre># Residuals are what the decomposition couldn&#39;t explain<br>features[&#39;stl_residual&#39;] = stl.resid<br><br># Rolling stats of residuals capture regime changes<br>features[&#39;residual_volatility&#39;] = stl.resid.shift(1).rolling(7).std()</pre><p>Sudden shifts in residual volatility often signal structural changes in the series — a new competitor entering the market, an operational disruption, a demand regime shift. Tree-based models can pick these up when the raw signal is too noisy.</p><h3>Decomposition as the Foundation of Anomaly Detection</h3><p>The residual component of a decomposition has a beautiful property: if the decomposition captured the real structure of the series, <strong>the residual should be white noise with roughly constant variance.</strong> Anything that isn’t — spikes, volatility clusters, extended runs above or below zero — is a candidate anomaly.</p><pre># STL-based anomaly detection<br>stl = STL(sales, period=7, robust=True).fit()<br>residuals = stl.resid<br><br># Establish normal residual distribution<br>threshold_upper = residuals.mean() + 3 * residuals.std()<br>threshold_lower = residuals.mean() - 3 * residuals.std()<br><br>anomalies = residuals[(residuals &gt; threshold_upper) | (residuals &lt; threshold_lower)]<br>print(f&quot;Detected {len(anomalies)} anomalies out of {len(residuals)} observations&quot;)<br>print(f&quot;Anomaly rate: {len(anomalies) / len(residuals):.2%}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SlRFvGBQ-7WUhEI5lwNihw.png" /><figcaption>Figure 6 — STL residual-based anomaly detection. The top panel shows the raw sales with detected anomalies in red. The bottom panel shows the residuals — the anomalies are exactly the points that exceed the ±3σ threshold (dashed red lines). The green shaded band represents the “normal” residual distribution.</figcaption></figure><p>This is a genuinely production-grade anomaly detector for time series with clear trend and seasonal structure. It’s used in industrial monitoring, retail demand tracking, and infrastructure observability. Facebook’s Prophet library is essentially a more elaborate version of this pattern.</p><p><strong>Two limitations worth knowing:</strong></p><ul><li>STL-based detection assumes the residuals are stationary and roughly Gaussian. Series with heavy-tailed residuals (financial returns, network traffic) violate this and need different distributional assumptions.</li><li>The threshold of 3σ is a convention, not a truth. For high-frequency operational data, you might tolerate 4σ. For safety-critical monitoring, 2σ. Pick the threshold based on the cost of false positives vs false negatives in your specific application.</li></ul><h3>Which Decomposition to Use When</h3><p>A practical decision framework based on what I actually reach for:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9slAHPqPMU4S77ZgbBcvxQ.png" /></figure><p>For a general starting point on unknown data: <strong>STL with </strong><strong>robust=True and </strong><strong>period set to your dominant seasonal cycle.</strong> It handles more edge cases correctly than any other single method.</p><h3>Key Takeaways</h3><ul><li>Decomposition separates a time series into <strong>trend</strong>, <strong>seasonality</strong>, and <strong>residual</strong> — the shared first step for forecasting, feature engineering, and anomaly detection</li><li><strong>Additive</strong> vs <strong>multiplicative</strong> models depend on whether seasonal amplitude scales with the series level — visual test on the raw plot decides it</li><li><strong>Classical decomposition</strong> is fine for tutorials but has real production limitations; <strong>STL with </strong><strong>robust=True</strong> is the modern default</li><li><strong>MSTL</strong> handles multiple seasonalities simultaneously — essential for retail (weekly + yearly) and energy (daily + weekly + yearly) data</li><li>Decomposition outputs make excellent features, but must be <strong>recomputed inside CV folds</strong> to stay leakage-safe</li><li><strong>STL residuals + 3σ thresholding</strong> is a production-grade anomaly detector for series with clear structure</li></ul><h3>What’s Next</h3><p>Article 4 — <em>“Missing Data in Time Series: Beyond Simple Imputation”</em> — covers what to do when your series has gaps. Forward-fill, backward-fill, linear interpolation, seasonal decomposition-based imputation, and the specific reasons why sklearn’s SimpleImputer is almost always wrong for time series.</p><p><strong>→ Subscribe to the newsletter to get Article 4 next Tuesday.</strong></p><blockquote><em>If you use decomposition in production for either forecasting or anomaly detection, drop a comment on what surprised you when you first shipped it. Every practitioner has a story where the residuals told them something the raw series didn&#39;t.</em></blockquote><blockquote>📚 <strong>Full companion code repository:</strong> <a href="https://github.com/Ameen2488/time-series-feature-engineering"><strong>https://github.com/Ameen2488/time-series-feature-engineering</strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f76b05f2bc9f" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/decomposing-time-series-separating-signal-from-noise-f76b05f2bc9f">Decomposing Time Series: Separating Signal from Noise</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Time Series Breaks Your ML Pipeline (And How to Fix It)]]></title>
            <link>https://blog.gopenai.com/why-time-series-breaks-your-ml-pipeline-and-how-to-fix-it-3177886faa54?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3177886faa54</guid>
            <category><![CDATA[demand-forecasting]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 11:58:00 GMT</pubDate>
            <atom:updated>2026-08-11T10:19:02.016Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Three lines of sklearn code that work perfectly on tabular data will silently destroy a time series model.</strong> This article covers what breaks, why it breaks, and the pipeline pattern that survives production.</blockquote><h3><strong>The Setup</strong></h3><p>In Article 1, we established that time series must be tabularized before ML algorithms can consume it. We built a feature matrix with lags, rolling statistics, and exogenous variables, trained a GradientBoostingRegressor, and got sensible predictions.</p><p>That was the easy part.</p><p>The moment you try to productionize that model — hyperparameter tuning, cross-validation, scaling features, forecasting multiple steps ahead — the standard sklearn workflow starts working <em>against</em> you. Not throwing errors. Just quietly returning results that look correct and aren’t.</p><p>This article covers the three failure modes that catch most practitioners, and the pipeline architecture that avoids them.</p><h3><strong>Failure #1 — The Temporal Train-Test Split</strong></h3><p>The most fundamental mistake in time series ML is using a standard random split.</p><pre># ❌ This will give you a beautiful, meaningless score<br>from sklearn.model_selection import train_test_split<br><br>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y, test_size=0.2, random_state=42<br>)</pre><p>Here’s what happens under the hood. train_test_split shuffles the observations before slicing them. On a time series with dates from January to December, a shuffled split puts a March observation in train and a February observation in test — then trains a model that has already seen &quot;future&quot; data when it needs to predict &quot;past&quot; data.</p><p>The result is a test score that looks strong in the notebook and collapses in production, because production has no future to peek at.</p><p><strong>The fix is deterministic slicing by time:</strong></p><pre># ✅ Split by position, preserving temporal order<br>split_idx = int(len(X) * 0.8)<br>X_train, X_test = X.iloc[:split_idx], X.iloc[split_idx:]<br>y_train, y_test = y.iloc[:split_idx], y.iloc[split_idx:]<br><br># Verify: last train date should be before first test date<br>print(f&quot;Last train date:  {X_train.index[-1]}&quot;)<br>print(f&quot;First test date:  {X_test.index[0]}&quot;)</pre><p>For cross-validation, use TimeSeriesSplit instead of KFold:</p><pre>from sklearn.model_selection import TimeSeriesSplit<br><br>tscv = TimeSeriesSplit(n_splits=5)<br><br>for fold, (train_idx, test_idx) in enumerate(tscv.split(X)):<br>    X_tr, X_te = X.iloc[train_idx], X.iloc[test_idx]<br>    y_tr, y_te = y.iloc[train_idx], y.iloc[test_idx]<br>    print(f&quot;Fold {fold}: train {len(X_tr)} rows → test {len(X_te)} rows&quot;)</pre><p>TimeSeriesSplit produces an expanding-window CV where each fold trains on all data up to a point and tests on the immediately following window. Training set grows, test set follows chronologically, no overlap.</p><p>For a more realistic evaluation, use <strong>walk-forward validation</strong> with a fixed training window instead of expanding — this simulates the production reality of retraining on a rolling window of recent data rather than all history.</p><pre>from sklearn.model_selection import TimeSeriesSplit<br><br># Fixed 500-row training window, 100-row test window, gap of 12 for horizon<br>tscv = TimeSeriesSplit(<br>    n_splits=5,<br>    max_train_size=500,   # rolling window (not expanding)<br>    test_size=100,        # fixed test size<br>    gap=12                # separate train and test by forecast horizon<br>)</pre><p>The gap parameter matters. If you&#39;re forecasting 12 steps ahead, the model can never see data from within 12 steps of what it&#39;s predicting — otherwise you&#39;re leaking near-future information into training.</p><h3><strong>Failure #2 — The Leaky Feature Pipeline</strong></h3><p>The second failure mode is subtler and much more common: preprocessing steps that peek across the train/test boundary.</p><p>The three classic offenders:</p><p><strong>Scalers fit on the full dataset</strong></p><pre># ❌ StandardScaler now knows the mean/std of the test set<br>scaler = StandardScaler()<br>X_all_scaled = scaler.fit_transform(X)  # &lt;-- fits on train + test<br>X_train, X_test = X_all_scaled[:split], X_all_scaled[split:]</pre><p>The scaler’s mean and standard deviation are computed using test observations, which means training features are normalized against statistics that include the future. In production, this is impossible — you don’t know tomorrow’s mean.</p><p><strong>Imputation with global statistics</strong></p><pre># ❌ Same problem, different transform<br>X_imputed = X.fillna(X.mean())  # mean includes test period</pre><p><strong>Rolling features computed on the full series</strong></p><pre># ❌ At time t, rolling(7) window includes y(t) itself<br>df[&#39;rolling_mean&#39;] = series.rolling(7).mean()</pre><p>We covered this in Article 1 — the fix is series.shift(1).rolling(7).mean(). But the pattern is worth restating: <strong>any transformation that uses information from future rows to compute values for current rows is a leak.</strong></p><h3>The fix: sklearn Pipelines</h3><p>Sklearn’s Pipeline object solves the scaler and imputer problem elegantly. It ensures every preprocessing step is .fit() on the training fold only, then .transform() applied to test data.</p><pre>from sklearn.pipeline import Pipeline<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.impute import SimpleImputer<br>from sklearn.ensemble import GradientBoostingRegressor<br><br>pipe = Pipeline([<br>    (&#39;impute&#39;,  SimpleImputer(strategy=&#39;median&#39;)),<br>    (&#39;scale&#39;,   StandardScaler()),<br>    (&#39;model&#39;,   GradientBoostingRegressor(n_estimators=200, max_depth=4)),<br>])<br><br># Now fit_transform is applied fold-safely inside cross_val_score<br>from sklearn.model_selection import cross_val_score<br><br>scores = cross_val_score(<br>    pipe, X, y,<br>    cv=TimeSeriesSplit(n_splits=5),<br>    scoring=&#39;neg_mean_absolute_error&#39;<br>)<br>print(f&quot;CV MAE: {-scores.mean():.2f} ± {scores.std():.2f}&quot;)</pre><p>For time-series-specific feature engineering — lag creation, rolling windows, seasonal decomposition — sklearn’s built-in transformers won’t cut it. The <a href="https://feature-engine.trainindata.com/">feature-engine</a> library provides drop-in LagFeatures, WindowFeatures, and ExpandingWindowFeatures transformers that follow the sklearn API and slot cleanly into pipelines.</p><pre>from feature_engine.timeseries.forecasting import (<br>    LagFeatures, WindowFeatures<br>)<br><br>pipe = Pipeline([<br>    (&#39;lags&#39;,    LagFeatures(variables=[&#39;sales&#39;], periods=[1, 7, 14])),<br>    (&#39;window&#39;,  WindowFeatures(<br>        variables=[&#39;sales&#39;], <br>        window=7, <br>        functions=[&#39;mean&#39;, &#39;std&#39;],<br>        freq=&#39;1D&#39;<br>    )),<br>    (&#39;impute&#39;,  SimpleImputer(strategy=&#39;median&#39;)),<br>    (&#39;scale&#39;,   StandardScaler()),<br>    (&#39;model&#39;,   GradientBoostingRegressor(n_estimators=200, max_depth=4)),<br>])</pre><p>This is the pattern I’ve deployed in production. Every transform respects fold boundaries, every feature computation is temporally safe, and the whole thing serializes cleanly for MLflow logging.</p><h3><strong>Failure #3 — The Multi-Step Forecast Trap</strong></h3><p>The third failure mode is architectural rather than mechanical. Everything above assumes you’re forecasting <strong>one step ahead</strong>. Real forecasting problems rarely stop there.</p><p>If you need to forecast 12 weeks ahead, you have three fundamentally different strategies, and choosing the wrong one destroys the model’s accuracy in ways cross-validation won’t always catch.</p><h3>Strategy 1 — Recursive</h3><p>Train a single one-step model, then feed its predictions back as inputs for the next step:</p><pre>def recursive_forecast(model, last_known_values, n_steps, lag_periods):<br>    &quot;&quot;&quot;<br>    Forecast n_steps ahead by feeding predictions back as lag inputs.<br>    &quot;&quot;&quot;<br>    forecasts = []<br>    history = list(last_known_values)  # rolling buffer of recent values<br>    <br>    for step in range(n_steps):<br>        # Build feature vector from current history<br>        features = [history[-lag] for lag in lag_periods]<br>        # Predict one step<br>        pred = model.predict([features])[0]<br>        forecasts.append(pred)<br>        # Append prediction to history for next step<br>        history.append(pred)<br>    <br>    return forecasts</pre><p><strong>Trade-off:</strong> Simple, only one model to train and maintain. But errors compound — the 12th step is built on 11 previously predicted values, each with its own error. Variance explodes with horizon.</p><h3>Strategy 2 — Direct</h3><p>Train one separate model per forecast step:</p><pre>from sklearn.multioutput import MultiOutputRegressor<br><br># Create separate targets for each horizon<br>def build_direct_targets(series, horizons):<br>    targets = pd.DataFrame({<br>        f&#39;y_h{h}&#39;: series.shift(-h) for h in horizons<br>    })<br>    return targets.dropna()<br><br>y_direct = build_direct_targets(sales, horizons=range(1, 13))<br><br># Train one model per horizon<br>model = MultiOutputRegressor(<br>    GradientBoostingRegressor(n_estimators=100, max_depth=4)<br>)<br>model.fit(X, y_direct)</pre><p><strong>Trade-off:</strong> Each model specializes in its horizon — no error compounding. But you now maintain 12 models, and each is trained on less data (later horizons need .shift(-h) which drops rows from the tail).</p><h3>Strategy 3 — Multi-Output (Vector Autoregression style)</h3><p>A single model with 12 output nodes, jointly predicting all horizons:</p><pre>from sklearn.ensemble import RandomForestRegressor<br><br># RandomForest natively supports multi-output regression<br>model = RandomForestRegressor(n_estimators=200, max_depth=6)<br>model.fit(X, y_direct)  # y_direct is (n_samples, 12)<br><br>predictions = model.predict(X_test)  # shape: (n_test, 12)</pre><p><strong>Trade-off:</strong> One model, captures inter-step correlations, but many tree-based libraries (GradientBoostingRegressor, LightGBM in standard mode) don&#39;t natively support multi-output — you&#39;d wrap them in MultiOutputRegressor which is functionally equivalent to Strategy 2.</p><h3>Which strategy to use?</h3><p>There is no universal answer. In production, the choice comes down to horizon length, retraining budget, and the shape of your error tolerance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ilt3nfN9gDe4XKW685DrBA.png" /></figure><p>The libraries <a href="https://skforecast.org/">skforecast</a> and <a href="https://nixtlaverse.nixtla.io/mlforecast/">mlforecast</a> provide all three strategies out of the box with clean APIs — worth learning if you work in forecasting regularly.</p><h3>Putting It All Together — A Production-Grade Setup</h3><p>Here’s the full pattern that survives real deployment: a sklearn Pipeline with feature-engine transformers, evaluated with walk-forward CV, using direct multi-horizon forecasting with a gap.</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.pipeline import Pipeline<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.impute import SimpleImputer<br>from sklearn.multioutput import MultiOutputRegressor<br>from sklearn.model_selection import TimeSeriesSplit, cross_val_score<br>from feature_engine.timeseries.forecasting import LagFeatures, WindowFeatures<br>from lightgbm import LGBMRegressor<br><br># 1. Build multi-horizon targets<br>horizons = list(range(1, 13))<br>y_multi = pd.DataFrame({<br>    f&#39;h_{h}&#39;: sales.shift(-h) for h in horizons<br>}).dropna()<br><br>X = sales.to_frame(name=&#39;sales&#39;).loc[y_multi.index]<br><br># 2. Assemble leakage-safe pipeline<br>pipe = Pipeline([<br>    (&#39;lags&#39;,    LagFeatures(variables=[&#39;sales&#39;], periods=[1, 2, 3, 7, 14, 28])),<br>    (&#39;window&#39;,  WindowFeatures(<br>        variables=[&#39;sales&#39;],<br>        window=[7, 14],<br>        functions=[&#39;mean&#39;, &#39;std&#39;],<br>        freq=&#39;1D&#39;<br>    )),<br>    (&#39;impute&#39;,  SimpleImputer(strategy=&#39;median&#39;)),<br>    (&#39;scale&#39;,   StandardScaler()),<br>    (&#39;model&#39;,   MultiOutputRegressor(<br>        LGBMRegressor(n_estimators=300, learning_rate=0.05, max_depth=6, verbose=-1)<br>    )),<br>])<br><br># 3. Walk-forward cross-validation with gap<br>tscv = TimeSeriesSplit(<br>    n_splits=5,<br>    max_train_size=730,   # 2 years of daily data<br>    test_size=90,         # 3 months<br>    gap=12                # match longest horizon<br>)<br><br># 4. Evaluate per-horizon<br>from sklearn.metrics import mean_absolute_error<br><br>mae_by_horizon = np.zeros(len(horizons))<br>for train_idx, test_idx in tscv.split(X):<br>    X_tr, X_te = X.iloc[train_idx], X.iloc[test_idx]<br>    y_tr, y_te = y_multi.iloc[train_idx], y_multi.iloc[test_idx]<br>    <br>    pipe.fit(X_tr, y_tr)<br>    y_pred = pipe.predict(X_te)<br>    <br>    for i, h in enumerate(horizons):<br>        mae_by_horizon[i] += mean_absolute_error(y_te.iloc[:, i], y_pred[:, i])<br><br>mae_by_horizon /= tscv.get_n_splits()<br><br># 5. Report degradation with horizon<br>for h, mae in zip(horizons, mae_by_horizon):<br>    print(f&quot;Horizon {h:2d}: MAE = {mae:.3f}&quot;)</pre><p>The output — MAE per horizon — is one of the most important diagnostics in production forecasting. It tells you <em>how much</em> your model’s accuracy degrades as you look further ahead, which is what stakeholders actually care about. A model with MAE of 5 at horizon 1 and 45 at horizon 12 might be perfectly usable for two weeks and useless past that. Cross-validated aggregate MAE hides this pattern completely.</p><h3>Key Takeaways</h3><ul><li>Standard train_test_split and KFold will silently destroy your time series model — always split by time</li><li><strong>Sklearn </strong><strong>Pipeline</strong> is not optional in time series ML; it&#39;s the only mechanism that guarantees preprocessing respects fold boundaries</li><li><strong>feature-engine</strong> and <strong>skforecast</strong> are the two libraries worth learning next — they extend sklearn’s grammar into time series feature engineering and forecasting strategies</li><li>Multi-step forecasting has three architectural strategies (recursive, direct, multi-output); the choice depends on horizon length and error tolerance, not personal preference</li><li>Always report <strong>MAE per horizon</strong>, not just aggregate MAE — the degradation curve is the actual diagnostic</li></ul><h3>What’s Next</h3><p>Article 3 — <em>“Decomposing Time Series: Separating Signal from Noise”</em> — covers the classical toolkit for understanding what’s inside your series before you engineer features on top of it. Trend extraction, seasonality detection, STL decomposition, and why decomposition is the first step in both forecasting and anomaly detection.</p><p><em>If you’ve built a time series pipeline in production, drop a comment with the leakage bug that took you longest to catch. The war stories are always the best teachers.</em></p><p>#TimeSeries #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #MLOps #Sklearn</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3177886faa54" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/why-time-series-breaks-your-ml-pipeline-and-how-to-fix-it-3177886faa54">Why Time Series Breaks Your ML Pipeline (And How to Fix It)</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tabularizing Time Series: The Foundation of ML-Based Forecasting]]></title>
            <link>https://blog.gopenai.com/tabularizing-time-series-the-foundation-of-ml-based-forecasting-2070d22651ef?source=rss-6e0470393cdb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2070d22651ef</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[time-series-data]]></category>
            <dc:creator><![CDATA[Ameen Siddiqui]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 11:59:26 GMT</pubDate>
            <atom:updated>2026-08-11T10:19:03.899Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Most ML tutorials treat time series as just another dataset. That assumption will silently destroy your model’s performance.</strong> Here’s the fundamental transformation every time series project needs before a single feature is engineered.</blockquote><h3><strong>The Core Problem</strong></h3><p>When you open a time series dataset, you typically see something like this:</p><pre>date          sales<br>2023-01-01    142<br>2023-01-02    157<br>2023-01-03    133<br>2023-01-04    189<br>...</pre><p>This is a <strong>sequential structure</strong>. Your ML model, whether it’s XGBoost, LightGBM, or a Linear Regression, expects a <strong>tabular structure</strong> — rows of independent observations, each with a set of features. These two structures are fundamentally incompatible unless you bridge them correctly.</p><p>That bridge is called <strong>tabularization</strong>, and getting it right is the single most important decision in a time series ML project.</p><h3><strong>What Tabularization Actually Does</strong></h3><p>Tabularization converts a 1D sequence of observations into a 2D matrix where:</p><ul><li>Each <strong>row</strong> represents a forecasting instance (a point in time)</li><li>Each <strong>column</strong> represents a feature derived from the history of the series</li></ul><p>The result looks like this:</p><pre>lag_1   lag_2   lag_3   lag_7   target<br>157     142     NaN     NaN     133<br>133     157     142     NaN     189<br>189     133     157     142     201<br>...</pre><p>Now XGBoost can learn that “if sales were high 7 days ago on the same weekday, tomorrow’s sales are likely high too.” Without tabularization, it simply cannot.</p><h3>Three Categories of Features in a Tabular Time Series</h3><h3>1. Features from the Target Variable (Endogenous)</h3><p>These are derived directly from the series you’re forecasting. They encode the <strong>historical behaviour</strong> of the target.</p><p><strong>Lag features</strong> are the most fundamental: y(t-1), y(t-2), ..., y(t-k) become columns in your feature matrix.</p><pre>import pandas as pd<br>import numpy as np<br><br>def create_lag_features(series: pd.Series, lags: list) -&gt; pd.DataFrame:<br>    &quot;&quot;&quot;<br>    Transform a univariate time series into a supervised learning<br>    dataset by creating lag features.<br>    <br>    Parameters<br>    ----------<br>    series : pd.Series with DatetimeIndex<br>    lags   : list of integers, e.g. [1, 2, 3, 7, 14]<br>    <br>    Returns<br>    -------<br>    pd.DataFrame with lag columns and a &#39;target&#39; column<br>    &quot;&quot;&quot;<br>    df = pd.DataFrame({&#39;target&#39;: series})<br>    for lag in lags:<br>        df[f&#39;lag_{lag}&#39;] = series.shift(lag)<br>    return df.dropna()<br><br><br># Example: daily sales data<br>np.random.seed(42)<br>dates = pd.date_range(&#39;2023-01-01&#39;, periods=100, freq=&#39;D&#39;)<br>sales = pd.Series(<br>    100 + np.cumsum(np.random.randn(100)) + 10 * np.sin(np.arange(100) * 2 * np.pi / 7),<br>    index=dates,<br>    name=&#39;sales&#39;<br>)<br><br>df = create_lag_features(sales, lags=[1, 2, 3, 7, 14])<br>print(df.head())</pre><p>Output:</p><pre>target      lag_1      lag_2      lag_3      lag_7     lag_14<br>2023-01-15  105.32     103.81     102.45     108.23      99.67      88.41<br>2023-01-16  108.91     105.32     103.81     102.45     103.12      91.23<br>...</pre><p><strong>Why these lags?</strong> Always start with domain knowledge:</p><ul><li>lag_1, lag_2, lag_3 → short-term autocorrelation</li><li>lag_7 → weekly seasonality</li><li>lag_14, lag_28 → bi-weekly / monthly patterns</li></ul><h3>2. Features from Exogenous Variables</h3><p>Many forecasting problems involve additional signals beyond the target itself. In retail demand forecasting, for example, promotions, holidays, and pricing are exogenous variables — they influence sales but aren’t derived from sales.</p><pre># Adding exogenous variables to the feature matrix<br>df_exog = pd.DataFrame({<br>    &#39;target&#39;: sales,<br>    &#39;is_weekend&#39;: (sales.index.dayofweek &gt;= 5).astype(int),<br>    &#39;is_holiday&#39;: [0]*90 + [1]*10,  # simplified<br>    &#39;price_index&#39;: np.random.uniform(0.9, 1.1, 100),<br>}, index=dates)<br><br># Now add lags of the target on top<br>for lag in [1, 7]:<br>    df_exog[f&#39;lag_{lag}&#39;] = sales.shift(lag)<br><br>df_exog = df_exog.dropna()</pre><p><strong>Critical distinction:</strong> Exogenous variables at forecast time t are <strong>future-known</strong> (you know if tomorrow is a holiday) versus <strong>future-unknown</strong> (you don&#39;t know tomorrow&#39;s actual price if it fluctuates). This distinction has enormous implications for your feature pipeline — a confusion many practitioners miss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lj-T85LCxYhq5UHDjDkjAg.jpeg" /></figure><h3>3. Window/Aggregate Features from History</h3><p>Rather than raw lag values, you can encode <strong>statistical summaries</strong> of recent history:</p><pre># Rolling window statistics<br>df[&#39;rolling_mean_7&#39;]  = sales.shift(1).rolling(7).mean()<br>df[&#39;rolling_std_7&#39;]   = sales.shift(1).rolling(7).std()<br>df[&#39;rolling_max_7&#39;]   = sales.shift(1).rolling(7).max()<br>df[&#39;expanding_mean&#39;]  = sales.shift(1).expanding().mean()</pre><p>Notice the .shift(1) before every window operation. <strong>This is not optional.</strong> Without it, your rolling mean at time t includes y(t) itself — a target leak that will give you suspiciously perfect training scores and catastrophically wrong test scores.</p><h3>Single-Step Forecasting: The Full Pipeline</h3><p>Putting it together, here is a complete single-step forecasting setup:</p><pre>from sklearn.ensemble import GradientBoostingRegressor<br>from sklearn.model_selection import TimeSeriesSplit<br>from sklearn.metrics import mean_absolute_error<br>import matplotlib.pyplot as plt<br><br># 1. Build the feature matrix<br>def build_feature_matrix(series: pd.Series) -&gt; pd.DataFrame:<br>    df = pd.DataFrame({&#39;target&#39;: series})<br>    <br>    # Lag features (shift ensures no leakage)<br>    for lag in [1, 2, 3, 7, 14]:<br>        df[f&#39;lag_{lag}&#39;] = series.shift(lag)<br>    <br>    # Window features (shift(1) before rolling = no leakage)<br>    s = series.shift(1)<br>    df[&#39;roll_mean_7&#39;]  = s.rolling(7).mean()<br>    df[&#39;roll_std_7&#39;]   = s.rolling(7).std()<br>    df[&#39;expand_mean&#39;]  = s.expanding().mean()<br>    <br>    # Datetime features (always available, no leakage)<br>    df[&#39;dayofweek&#39;]    = series.index.dayofweek<br>    df[&#39;month&#39;]        = series.index.month<br>    <br>    return df.dropna()<br><br>df_features = build_feature_matrix(sales)<br><br>X = df_features.drop(columns=&#39;target&#39;)<br>y = df_features[&#39;target&#39;]<br><br># 2. Time-aware train-test split (NEVER shuffle!)<br>split_idx = int(len(X) * 0.8)<br>X_train, X_test = X.iloc[:split_idx], X.iloc[split_idx:]<br>y_train, y_test = y.iloc[:split_idx], y.iloc[split_idx:]<br><br># 3. Train<br>model = GradientBoostingRegressor(n_estimators=200, max_depth=4, random_state=42)<br>model.fit(X_train, y_train)<br><br># 4. Evaluate<br>y_pred = model.predict(X_test)<br>mae = mean_absolute_error(y_test, y_pred)<br>print(f&quot;Test MAE: {mae:.2f}&quot;)<br><br># 5. Visualize<br>plt.figure(figsize=(12, 4))<br>plt.plot(y_test.index, y_test.values, label=&#39;Actual&#39;, linewidth=2)<br>plt.plot(y_test.index, y_pred, label=&#39;Predicted&#39;, linestyle=&#39;--&#39;, linewidth=2)<br>plt.title(&#39;Single-Step Forecast: GBM on Tabularized Time Series&#39;)<br>plt.legend()<br>plt.tight_layout()<br>plt.savefig(&#39;article1_forecast.png&#39;, dpi=150)<br>plt.show()</pre><h3>The Critical Pitfall: Temporal Leakage</h3><p>This is the mistake that makes models look brilliant in development and useless in production.</p><p><strong>Leakage occurs when information from the future implicitly enters your features.</strong> In time series, the three most common sources are:</p><ol><li><strong>Rolling statistics without shift</strong> — series.rolling(7).mean() at time t includes y(t)</li><li><strong>Standard train-test split</strong> — train_test_split(shuffle=True) mixes future into training</li><li><strong>Normalization/scaling on the full dataset</strong> — fitting a StandardScaler on all data before splitting leaks test statistics into training</li></ol><pre># ❌ WRONG: This leaks y(t) into the feature<br>df[&#39;roll_mean&#39;] = series.rolling(7).mean()<br><br># ✅ CORRECT: Shift by 1 ensures feature at t uses only t-1, t-2, ..., t-7<br>df[&#39;roll_mean&#39;] = series.shift(1).rolling(7).mean()<br><br># ❌ WRONG: Random split destroys temporal ordering<br>from sklearn.model_selection import train_test_split<br>X_train, X_test = train_test_split(X, test_size=0.2, shuffle=True)<br><br># ✅ CORRECT: Always split by time<br>split = int(len(X) * 0.8)<br>X_train, X_test = X.iloc[:split], X.iloc[split:]</pre><h3>Connecting to Anomaly Detection</h3><p>Everything above applies directly to anomaly detection in time series. The logic is the same: you tabularize the series into a feature matrix, but instead of predicting the next value, you model the <strong>expected behaviour</strong> and flag points where the prediction error exceeds a threshold.</p><pre># Anomaly detection via reconstruction error<br>y_pred_train = model.predict(X_train)<br>residuals_train = y_train - y_pred_train<br><br># Establish normal error distribution on training data<br>threshold = residuals_train.mean() + 3 * residuals_train.std()<br><br>y_pred_all = model.predict(X)<br>residuals_all = y - y_pred_all<br><br>anomalies = residuals_all[residuals_all.abs() &gt; threshold]<br>print(f&quot;Detected {len(anomalies)} anomalies&quot;)</pre><p>This simple approach — <strong>fit → predict → threshold residuals</strong> — is the backbone of many production anomaly detection systems.</p><h3>Key Takeaways</h3><ul><li>Time series data must be <strong>tabularized</strong> before any standard ML algorithm can consume it</li><li>The feature matrix consists of <strong>lag features</strong>, <strong>window features</strong>, and <strong>exogenous variables</strong></li><li><strong>Temporal leakage</strong> is the #1 silent killer — always shift before rolling, always split by time</li><li>Single-step forecasting is the entry point; multi-step and recursive forecasting (Article 2) adds significant complexity</li><li>The same tabularization framework powers both <strong>forecasting</strong> and <strong>anomaly detection</strong></li></ul><h3>What’s Next</h3><p>In Article 2 — <em>“Why Time Series Breaks Your ML Pipeline (And How to Fix It)”</em> — we’ll go deeper into the challenges that tabularization introduces: proper cross-validation for time series, building sklearn-compatible pipelines that avoid leakage, and the critical difference between direct, recursive, and multi-output forecasting strategies.</p><p><em>If you found this useful, leave a comment below — especially if you’ve been burned by temporal leakage in production. I’d love to hear the war story.</em></p><p>#TimeSeries #Forecasting #FeatureEngineering #DataScience #MachineLearning #Python #AnomalyDetection</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2070d22651ef" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/tabularizing-time-series-the-foundation-of-ml-based-forecasting-2070d22651ef">Tabularizing Time Series: The Foundation of ML-Based Forecasting</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>