<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Piotr Kalanski on Medium]]></title>
        <description><![CDATA[Stories by Piotr Kalanski on Medium]]></description>
        <link>https://medium.com/@piotr.kalanski?source=rss-4504af9f33e0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VUvkfJS39xCQfMrtj7vE3w.jpeg</url>
            <title>Stories by Piotr Kalanski on Medium</title>
            <link>https://medium.com/@piotr.kalanski?source=rss-4504af9f33e0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:35:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@piotr.kalanski/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your Supply Forecast Is a Roster Count in Disguise]]></title>
            <link>https://medium.com/@piotr.kalanski/your-supply-forecast-is-a-roster-count-in-disguise-8e908d030cc5?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e908d030cc5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gig-economy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 13 Nov 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-11-13T07:01:02.171Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0xlSz-t-YUYP84CxIuF-pw.png" /><figcaption>source: Generated by author using ChatGPT</figcaption></figure><p><em>Why counting active workers overstates availability, and what to build instead</em></p><p>We cut the lookback window on our supply estimate from 36 months to 18, and more than a third of our “available” workers vanished. Nothing about the worker pool changed. The wider window had been overstating available supply by roughly 59%, counting people who hadn’t taken an assignment in over eighteen months and, in most cases, were never coming back.</p><p>We found this while building a demand-supply forecasting system for a temporary staffing marketplace, chasing a gap signal that kept underestimating shortfalls. The model was doing its job on a bad input. A roster count answers “who have we seen recently?” A supply forecast has to answer “who will be available to work eight weeks from now?” Those are different questions, and conflating them misdirects recruiting effort.</p><p>This article walks through how we replaced the count with a probabilistic estimate: a calibrated churn model, eligibility filters, an onboarding pipeline tier, and a decomposition that turns one number into a diagnostic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2esvpbd8G203Mhz0dzAtTQ.png" /><figcaption>End-to-end supply estimation flow: Tier 1 active workers pass eligibility gates and a calibrated churn model to become summed survival probabilities, Tier 2 onboarding-pipeline supply joins them, and total expected supply feeds the demand gap and recruitment priority list</figcaption></figure><h3>Active Roster ≠ Available Supply</h3><p>The naive supply estimate is easy to build, which is why most systems start there: count workers dispatched within the last N months, aggregated by geography, job type, and week. Ours started there too, with the 36-month window as the default assumption in the initial design.</p><p>It fails for a structural reason: temporary staffing has high worker turnover. A meaningful fraction of workers who look active today will not take an assignment in the next eight weeks. The roster is a snapshot of the past, and supply forecasting needs a prediction about the future.</p><p>The error compounds downstream. The gap signal is demand minus supply, and it drives recruitment prioritization. Overstated supply understates the gap, recruitment gets under-allocated, and shortfalls arrive unexpectedly in zones the system claimed were healthy.</p><p>Tightening the window helps, and we validated the ~59% overstatement against dispatch history before changing it. But no window solves the underlying problem. An 18-month roster still treats a worker with a 50% chance of leaving as a full unit of supply. The fix is to replace the count with a probability.</p><h3>Expected Supply as a Sum of Survival Probabilities</h3><p>The core of the supply estimate is one formula:</p><pre>expected_supply(zone, job_duty, horizon_week) = Σ survival_probability_i</pre><p>summed over every worker <em>i</em> who is active within the validated availability window, has history in the job duty, and has cleared a background check. (Zones here are <a href="https://medium.com/@piotr.kalanski/your-demand-forecast-is-using-the-wrong-map-0f18ec53773b">commute zones</a> derived from dispatch patterns rather than cities.)</p><p>The survival probability for worker <em>i</em> is the probability that they are still active and dispatchable at week W+N. A worker with 0.8 survival probability contributes 0.8 to expected supply, not 1.0. Sum those probabilities across a zone and you get expected available headcount, fractional churn risk included.</p><p>This framing imposes a requirement that’s easy to miss: calibration matters as much as discrimination. AUC tells you whether the model ranks high-churn workers above low-churn workers. Brier score tells you whether a predicted 0.8 corresponds to 80% of those workers actually surviving. When you sum probabilities into headcount, calibration is the supply accuracy lever. A model with AUC 0.90 and poor calibration will overstate or understate supply for every zone, every week, in the same direction.</p><p>We set Brier ≤ 0.15 as the gate and report both metrics together.</p><h3>The Churn Model: What to Predict and How to Label It</h3><p>The probabilities come from a binary classifier: will this worker have churned within N weeks? We built the first version of this worker churn model in 2023; the forecasting system extends it to multiple horizons — W+4, W+6, W+8, and W+12 — covering the range of recruitment lead times. The output must be a calibrated probability, not a score; if the base model isn’t well-calibrated, add Platt scaling or isotonic regression as a post-processing step.</p><p>The label is where the domain knowledge lives. The naive label says: no dispatches during the horizon window means churned. The problem is that temporary workers take breaks. Seasonal gaps, family obligations, a paused project. Label those workers churned and the model learns to suppress supply estimates for anyone with irregular patterns.</p><p>The fix is a confirmation window. A worker is labeled churned only if they had no dispatches during the horizon <em>and</em> no dispatches in the six weeks after it. The confirmation window separates true departure from temporary inactivity. It’s a label engineering decision, not a tuning parameter, and getting it wrong can’t be repaired later by calibration.</p><p>Features, ranked by SHAP importance:</p><ul><li>Recency: weeks and days since last dispatch, the dominant predictors</li><li>Activity volume: dispatch counts over the last 4 and 13 weeks</li><li>Tenure: length of tenure, in weeks</li><li>Reliability: completion rate and call-off rate</li><li>Calendar: week of year, which captures seasonal patterns</li></ul><p>Training is a LightGBM classifier with balanced class weights, because churn is the minority class inside an active worker pool. Validation uses a temporal split: train on weeks 1 through N, validate on the weeks after. Never split randomly when labels are derived from future windows; the leakage is invisible and flattering.</p><p>On held-out temporal validation at the 8-week horizon, the model we’re taking into pilot reaches AUC 0.87, F1 0.80, and Brier 0.13. The Brier score is the number that matters most here: below the 0.15 gate, the probabilities are reliable enough to sum into headcount.</p><h3>Not Every Active Worker Is Available for Every Role</h3><p>Survival probability is necessary but not sufficient. A worker who will still be active in eight weeks is not supply for a forklift role if they’ve never operated a forklift. Two hard eligibility gates apply before any probability enters the sum.</p><p>The first is job duty history. A worker counts toward supply for a job duty only if they’ve been dispatched to that duty type within the past few years, resolved through a feature store lookup. This matters most for specialized roles like CDL drivers or forklift operators, where supply is genuinely constrained and over-counting hurts the most.</p><p>The second is background check status, a binary gate: cleared or not. A worker who hasn’t cleared one cannot be dispatched regardless of skill or intent, so only cleared workers contribute to eligible supply. The status already existed in our feature store, so the filter was nearly free to add for a meaningful precision gain.</p><p>One implementation note that pays off later: compute supply at each gate and keep the intermediate numbers. We apply the duty filter first, so the raw pool is already duty-specific; the step from raw to eligible then isolates the background check effect. That intermediate delta becomes the backbone of the diagnostic below.</p><h3>The Supply Your Roster Doesn’t Know About</h3><p>Everything so far covers workers who have already been dispatched at least once. That misses a category the roster cannot see: candidates currently in the onboarding pipeline who will become dispatchable within the forecast horizon. For longer horizons, W+8 and W+12, this Tier 2 supply is where the forward-looking signal lives.</p><p>The calculation has three inputs:</p><pre>tier2_supply(zone, job_duty, horizon) =<br>  pipeline_candidates(zone)<br>  × conversion_rate(zone, horizon)<br>  × first_dispatch_distribution(zone, job_duty)</pre><p>Pipeline candidates come from ATS and onboarding data: candidates mapped from zip code to commute zone who haven’t had a first dispatch. In our dataset, zip coverage was ~90% and job category coverage ~89%, enough for zone-level estimates. Conversion rates are computed as monthly cohorts: of the candidates who started onboarding in month M, what share had a first dispatch within N weeks? The first-dispatch distribution converts expected headcount into supply per job duty, based on which duties newly onboarded workers historically take. Where a zone’s sample is too sparse for stable rates, fall back to a broader geography average.</p><p>Tier 2 is also the most fragile component of the estimate, and the output should say so. Conversion rates are noisy in thin zones and go stale when onboarding processes change. Before the rates feed the pilot, we required a cohort stability check: variance across a 12-month window had to stay within bounds. Treat Tier 2 as a forward-looking signal behind a validation gate, weaker evidence than Tier 1.</p><h3>One Number Hides the Problem; Three Layers Reveal It</h3><p>Total supply is the output the gap calculation needs, but the layered view is what makes it actionable:</p><ul><li><strong>Tier 1, raw:</strong> active workers in the window with matching duty history. This is the baseline duty-specific pool.</li><li><strong>Tier 1, eligible:</strong> the background-check-cleared subset. The drop from raw to eligible is the background check gap.</li><li><strong>Tier 1 eligible + Tier 2:</strong> the full forward-looking supply. This is the number demand is compared against.</li></ul><p>When total supply falls short of demand, the layers say why. High raw Tier 1 with low eligible Tier 1 points at a background check backlog, and recruiting won’t fix that; faster check processing will. Adequate eligible Tier 1 with thin Tier 2 means the onboarding pipeline is dry: a recruiting gap. Both tiers thin together means churn is the driver, and no amount of recruiting outruns a retention problem cheaply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9Yr0XzhmkpFgKkg1VKcWg.png" /><figcaption>Three shortfall patterns in the layered supply decomposition — a background check backlog, a dry onboarding pipeline, and churn-driven thinning across all layers — each pointing to a different fix</figcaption></figure><p>One honest limit: when every layer is thin at once, the decomposition alone won’t rank the fixes. Pair it with the demand trend to decide what to attack first.</p><p>A recruiter reading this output doesn’t just learn that a gap exists. They learn what kind of gap it is, and the kind dictates the action.</p><h3>Connecting Supply to the Recruitment Priority List</h3><p>The last step is mechanical. The gap is the demand forecast minus total supply, and urgency weights nearer horizons higher because there’s less time to act:</p><pre>raw_gap = demand_forecast − total_supply<br><br>urgency = max(0, raw_gap) × lead_time_weight(horizon)</pre><p>The output is a recruitment priority list per zone, job duty, and horizon week, with urgency scores bucketed into high/medium/low for the recruiter-facing view. Two illustrative rows:</p><ul><li>Zone A, warehouse, W+4: demand 120, supply 84, gap 36, urgency high</li><li>Zone B, forklift, W+8: demand 45, supply 38, gap 7, urgency medium</li></ul><p>One architecture decision worth stating: supply estimation and gap scoring are separate pipelines. Supply logic keeps evolving, with better Tier 2 modeling and new eligibility layers such as drug test status. Gap logic is stable arithmetic. Coupling them would make the stable part expensive to change every time the volatile part improves.</p><h3>What Transfers</h3><p>Supply forecasting that counts workers instead of estimating their survival produces a gap signal that lies to you, and it lies in the worst direction: it hides shortfalls until they arrive.</p><p>The structure transfers to any labor platform with worker churn and an onboarding pipeline. Train a churn model and hold it to a calibration gate alongside AUC. Apply eligibility filters before summing. Add pipeline-based Tier 2 supply for longer horizons, with a stability check on its conversion rates. Keep the layered decomposition in the output, because the layers are what convert a forecast into a decision.</p><p>The open question as the system moves through its pilot is Tier 2 drift: how often conversion rates shift when onboarding processes change. It’s the highest-variance component of the estimate and the first place we expect to recalibrate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e908d030cc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Horizon Demand Forecasting: Why Architecture Beats Algorithms]]></title>
            <link>https://medium.com/@piotr.kalanski/multi-horizon-demand-forecasting-why-architecture-beats-algorithms-06532cf55822?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/06532cf55822</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[demand-forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 08:01:02 GMT</pubDate>
            <atom:updated>2026-10-30T08:01:02.593Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdxl8FWvH4h0yhT8JHzu4A.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>How we forecast at location × job type granularity, moved from Prophet to XGBoost, and why the hardest decisions had nothing to do with models.</blockquote><p>A recruiter doesn’t need to know that demand will be “broadly elevated” next quarter. They need to know that a specific location will be short on warehouse operatives in weeks 4 through 12, with enough lead time to act. The forecast that matters lives at location × job type, at the horizon where recruitment decisions are still reversible.</p><p>That specificity is what makes the problem hard. Location × job type intersections are sparse by nature. Many combinations have thin history or long runs with no demand at all, and the architecture has to accommodate that. When a forecast at that granularity drifts at 12 weeks, the cost arrives as unfilled shifts and lost revenue, weeks before anyone sees it in a metric.</p><p>Solving it comes down to one question, asked separately for every horizon: what information will exist when this forecast runs? Multi-horizon demand forecasting is a data contract problem, and the hardest decisions in this system were about that contract.</p><h3>The Business Problem That Shaped Every Design Decision</h3><p>Workforce supply platforms connect businesses with flexible workers. Fill rate, the share of open shifts covered, is a direct revenue signal. Unfilled shifts aren’t a service quality problem in the abstract; they’re a billing event that didn’t happen.</p><p>Proactive recruitment closes that gap, and recruitment has lead times. A signal that arrives a week before a shortage is too late to source anyone. The forecast needs to reach 4–12 weeks out, at location × job type level, reliably enough to act on.</p><p>The business sets that range. If recruiting for a role takes 6 weeks end-to-end, a 4-week forecast for that role is useless, so the horizon is fixed from the first design decision rather than tuned later like a hyperparameter.</p><p>Granularity is just as fixed. “New York will need more workers in Q3” doesn’t tell a recruiter where to focus or what to source. The location × job type intersection is the unit of decision, and it is also where the sparsity comes from. New combinations with almost no history and established ones that go quiet for months share a model with dense, well-behaved series.</p><h3>From Prophet to XGBoost: What Changed and Why</h3><p>Prophet was a reasonable first choice. It’s interpretable and handles seasonality out of the box, which matters when a system has to prove value quickly across many time series.</p><p>The ceiling became visible at scale. One Prophet model per combination learns only from its own history, which at sparse combinations is thin, with no way to borrow signal from similar ones. Prophet does accept extra regressors, but each one needs a known value for every future date, which rules out most of the lagged and cross-series signals that predict demand shifts. Its seasonal decomposition assumptions also don’t hold cleanly on irregular, sparse series.</p><p>We moved to gradient boosting, tried both XGBoost and LightGBM, saw similar performance, and run XGBoost in production. A single model learns across all combinations, with location and job type as categorical features, and lags, rolling windows, and exponentially weighted moving averages become first-class inputs. A surge in warehouse operative demand in one city often leads demand in adjacent cities by two to three weeks. A cross-series model can capture this; a siloed Prophet model cannot.</p><p>The result was a 14% MAE reduction at the combination level, which meant fewer mis-allocated recruitment efforts and a cleaner fill rate signal at the intersections that drive decisions. The gain came from architecture, from one model learning across series with signals that Prophet’s per-series design couldn’t use. XGBoost and LightGBM performing similarly points the same way. Once the model could learn across series, the choice of library made little difference.</p><p>It also moved the hard problem. A model that consumes lags and cross-series signals is only as good as the guarantee that those values exist when it runs, and that guarantee changes with every week you add to the horizon.</p><h3>What Actually Changes When You Add Horizons</h3><p>A working 1-step model is not a multi-horizon model with a configuration change. It’s the most common mistake when extending a single-horizon baseline, and more expensive than it looks.</p><p>The first change is the target. Each horizon needs its own shifted target, because forecasting week 8 and forecasting week 4 are different modeling problems even on the same series.</p><p>The second change is feature availability, and this is where a 1-step model hides a trap. Every lag counts back from an anchor, either the forecast origin (the last observed week when the forecast runs) or the target week being predicted. In a 1-step model the two coincide, since lag 1 of the target week is the latest observation at the origin. Add horizons and they come apart. A lag of k weeks counted back from the target week only exists when k is at least the horizon. The same lag_6 column is valid for the 4-week model, where it points two weeks into observed history, and leaks future demand into the 8- and 12-week models, where it points two and six weeks past the origin.</p><p>Feature code written for a 1-step baseline usually computes lags on the row being predicted, which is the target anchor. So the rule in this system is that every historical feature is anchored at the forecast origin. Computed that way, lags, rolling windows, and EWMA are valid at every horizon.</p><p>Two more details complete the contract. On sparse combinations, a positional lag_1 means “the previous row,” which can be months back, so origin-anchored history has to be computed on the calendar rather than by row position. That failure is the subject of my companion piece, <a href="https://medium.com/@piotr.kalanski/416c540b07c8">Time Series Feature Engineering: Why Positional Lags Silently Break on Sparse Data</a>. The origin itself also needs a precise definition. If ingestion lags, “last week’s filled shifts” isn’t there when the forecast runs, and production feeds the model older data than training assumed.</p><p>Sparsity raises the cost of every slip. A dense series can absorb a misaligned feature; at a sparse combination the error surfaces faster, often only in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-YHN2Rujwx1aFO-EhpO4Q.png" /><figcaption>Feature availability at inference time by horizon. Origin-anchored history is valid at H=4, H=8, and H=12; a lag anchored to the target week is valid only when its offset is at least the horizon</figcaption></figure><p>The third change is operational. Registry entries, deployment artifacts, monitoring dashboards, and retraining schedules all multiply by horizon count, which is manageable if it’s designed for upfront.</p><h3>Four Approaches, Ranked by Viability</h3><p>We compared four approaches on accuracy at long horizons, leakage risk, codebase impact, operational complexity, and fit with the business requirement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0uQ7kyEzOpvoR8rhvFs1tA.png" /><figcaption>Four multi-horizon forecasting approaches compared by viability for sparse location × job type forecasting at 4–12 week horizons</figcaption></figure><p><strong>Recursive forecasting</strong> uses a 1-step model and feeds its own predictions back as inputs for future steps. It’s intuitive and needs almost no upfront work, but error compounds. By week 12, the lags it consumes are several generations of its own predictions removed from anything observed, and sparse combinations have no stable signal to absorb that drift. Not viable beyond short horizons for this problem.</p><p><strong>Aggregate forecasting</strong> forecasts at region or job family level, then disaggregates. Useful for some planning questions, but it destroys the location × job type signal that makes the forecast actionable. Wrong tool for this problem.</p><p><strong>Multi-output models</strong> use one model with an output per horizon. Theoretically elegant, but tree-based models don’t share representations across outputs the way neural networks do. In its common forms (sklearn’s MultiOutputRegressor, or XGBoost’s default of one tree per output), multi-output gradient boosting is really multiple models with shared training logistics. Every output also sees the same inputs, so a column like the target-anchored lag_6 above, safe at 4 weeks and leaking at 12, can’t be scoped to the outputs it’s valid for, and offline metrics reward the leak instead of flagging it. Tooling and codebase impact are the highest of the four. A valid Phase 2 option for teams with mature infrastructure, but not the right starting point.</p><p><strong>Direct multi-horizon models</strong> use one model per horizon, each predicting exactly one future point. Each model gets a feature contract for its own horizon, and no prediction is ever fed back as an input, so error doesn’t accumulate. Adding a horizon means training one more model; existing pipelines barely change. The cost is operational overhead, since model count grows with horizon count. This is the right default.</p><h3>How Direct Multi-Horizon Models Work in Practice</h3><p>The core design is one horizon, one model. The week-8 model trains on targets shifted 8 weeks forward, the week-12 model on targets shifted 12, and each sees only features that exist at its forecast origin.</p><p>Pre-planned features such as public holidays, calendar events, and known contract cycles are anchored the other way from history, to the target week, because they are known in advance. They are among the highest-value inputs at long horizons, where recent history has less predictive pull. Cross-series features bridge sparsity gaps between combinations.</p><p>Implementation changes are contained. The dbt feature layer selects the correct pre-planned columns per horizon, so the contract is enforced before training sees the data. The training pipeline parameterizes target shifting by horizon. The model registry treats horizon as a first-class identifier. Core training abstractions don’t change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*915w-YxS_wsF0ZD6RH0COg.png" /><figcaption>Direct multi-horizon forecasting architecture: one XGBoost model per horizon, with explicit feature contracts from data sources through the model registry to predictions</figcaption></figure><p>Testing is not optional. Target shifting is the most failure-prone component in a multi-horizon system, and it fails silently. Consider a horizon-12 model accidentally trained on targets shifted by 4 weeks. Offline metrics look plausible, if anything better than a 12-week model should manage, because it is really a 4-week model scored against equally mis-shifted labels. In production, every surge it predicts for week 12 lands in week 4. Recruiters are still sourcing for week 12 when the demand arrives, and fill rate data shows it weeks later, after the shifts are lost.</p><p>The test that catches this is cheap. Build a synthetic series where demand equals the week index, run each horizon’s target builder, and assert that the label on week t equals t + H. A horizon-12 builder that shifts by 4 fails on the first row. Drop a few weeks from the series and the same test catches a positional shift that skips the gaps. At sparse combinations there is no population-level averaging to mask a systematic shift error, so a silent failure here is a business failure.</p><h3>What This Architecture Doesn’t Solve</h3><p>Direct multi-horizon models handle the architectural problem cleanly. Sparsity is a separate problem.</p><p>Gradient boosting cross-learns across combinations, which covers a lot of thin history, but it has limits. A new location or job type has no history of its own and only partial neighbors to borrow from, and no feature contract fixes that.</p><p>The features carry a sparsity trap of their own, the positional lags described earlier. The companion article covers how to audit for them and how calendar-aware transformers fix them inside an sklearn pipeline. This architecture is the foundation that feature work sits on.</p><h3>Conclusion</h3><p>The fastest way to lose trust in a forecasting system is to predict the future with information you won’t have at inference time. At sparse location × job type combinations, that failure shows up as unfilled shifts long before it registers as a metric regression.</p><p>Horizon-aware thinking is a production ML skill. Most teams spend months optimizing algorithms when the binding constraint was always the data contract.</p><p>Start with direct multi-horizon models: one model per horizon, history anchored at the forecast origin, calendar features anchored at the target week, and a test that proves every target is shifted by exactly its horizon. Get that right, and the recruiter acting on a week-8 signal is working from information that will exist when the forecast runs. That’s what closes the fill rate gap.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06532cf55822" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Demand Forecast Is Using the Wrong Map]]></title>
            <link>https://medium.com/@piotr.kalanski/your-demand-forecast-is-using-the-wrong-map-0f18ec53773b?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f18ec53773b</guid>
            <category><![CDATA[network-analysis]]></category>
            <category><![CDATA[demand-forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 07:01:03 GMT</pubDate>
            <atom:updated>2026-10-23T07:01:03.435Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dhqP87-PexfXS5xfdineFw.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>Replacing city-level proxies with data-driven commute zones for labor market forecasting</blockquote><p>You have a demand forecasting model for a staffing marketplace. It forecasts at city level. It holds up in large cities and breaks everywhere else. Small cities produce erratic predictions. Adjacent cities in the same metro behave as if they share nothing, even when they’re drawing from the same worker population. The cause, when you trace the failures, is that workers don’t respect city limits. When we hit this, we left the model alone and changed the geographic unit. With the same architecture and features, demand forecast accuracy moved from R² ~0.60 to ~0.89.</p><p>This article walks through a concrete approach for practitioners building labor market forecasting systems: derive commute zones from dispatch history using graph clustering, wire them into your forecasting pipeline, and measure the difference.</p><h3>The Problem with Forecasting at City Level</h3><p>Cities are administrative boundaries. Labor markets ignore them.</p><p>A worker who lives in one city regularly takes assignments in two or three adjacent ones. A large metro can span five to ten distinct municipalities, each appearing as a separate model unit despite sharing the same worker pool. City-level forecasting fragments this signal across units that have no behavioral independence.</p><p>Small cities are the worst case. Low weekly volume means high variance, which means your model has little trend to learn and a lot of noise to overfit. The model is learning spurious geography.</p><p>The failure that pushed us off city grain was a three-city metro where the model showed apparent worker surplus in one city and apparent shortage in the adjacent one. The same fifteen workers were driving both readings. The problem was the map, not the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IK5-hMsU2w2IQvnJr7gHBg.png" /><figcaption>City-level fragmentation vs. commute zone view — same workers appear as surplus in one city and shortage in another</figcaption></figure><p>On the demand side, training samples per city are too small for stable weekly forecasting, so variance dominates trend. On the supply side, workers who commute across city lines appear to be missing from adjacent cities. Both demand and supply are distorted by the same boundary mismatch, and the gap between them carries that error forward into operational decisions.</p><p>A bigger model or more features won’t fix this. The geographic unit has to be right first.</p><h3>What Commute Zones Are and How to Derive Them</h3><p>The core idea is to group ZIP codes that share overlapping worker pools. If a meaningful fraction of workers at ZIP A have also worked at ZIP B, the two ZIPs belong to the same labor market.</p><p><strong>Step 1 — Extract worker–ZIP pairs.</strong> From historical paid dispatches, compute distinct (worker, ZIP code) pairs over a two-year rolling window. Each worker belongs to every ZIP they’ve worked in. Filter to ZIPs with at least three unique workers and ten dispatches to avoid polluting the edge list with noise.</p><p><strong>Step 2 — Compute Jaccard similarity.</strong> For each pair of ZIPs sharing at least two workers:</p><pre>Jaccard(A, B) = |workers(A) ∩ workers(B)| / |workers(A) ∪ workers(B)|</pre><p>Jaccard is symmetric and normalized to 0–1, so raw ZIP volume alone doesn’t inflate an edge. High Jaccard means the ZIPs draw from the same labor pool. The result is a sparse edge list with ZIPs as nodes and Jaccard as edge weight.</p><p><strong>Step 3 — Louvain community detection.</strong> Drop edges below a Jaccard threshold (e.g.: 0.03) and build a weighted graph from what remains. Run Louvain, which partitions the graph into communities by maximizing modularity, grouping ZIPs that are tightly interconnected. Each community is a commute zone.</p><p><strong>Step 4 — Handle isolated ZIPs.</strong> ZIPs with no edges above the threshold get a fallback: city → county → singleton. Every ZIP gets a zone.</p><p><strong>Step 5 — Write the mapping.</strong> Output maps (location_key, effective_date) to commute_zone_id. Store it with an effective date so downstream models can do point-in-time joins against it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9k6ZqgwbOSUZroE-ISBvw.png" /><figcaption>Commute zone derivation pipeline — five steps from dispatch history to versioned zone mapping</figcaption></figure><p>The primary tuning levers are the Jaccard threshold (lower produces oversized zones, higher produces singletons) and Louvain resolution (higher means smaller, tighter zones). Target zones of five to thirty ZIPs with geographic coherence. Fewer than five usually means singletons aren’t clustering; more than thirty means the threshold is too permissive and geographically distant ZIPs are collapsing together.</p><h3>Building the Pipeline: dbt + Python</h3><p>The implementation splits across two stages: dbt handles the heavy SQL aggregation, Python handles graph clustering.</p><p><strong>dbt layer.</strong> Two models. The first, zip_dispatch_worker_pairs, produces one row per worker × ZIP pair from paid dispatches in the lookback window. The second, zip_worker_overlap, produces the graph edge list:</p><pre>WITH zip_stats AS (<br>  SELECT zip_code,<br>    COUNT(DISTINCT worker_number) AS unique_worker_count,<br>    SUM(dispatch_count) AS total_dispatch_count<br>  FROM zip_dispatch_worker_pairs<br>  GROUP BY zip_code<br>  HAVING COUNT(DISTINCT worker_number) &gt;= 3<br>         AND SUM(dispatch_count) &gt;= 10<br>),<br>shared_workers AS (<br>  SELECT<br>    a.zip_code AS zip_a,<br>    b.zip_code AS zip_b,<br>   COUNT(DISTINCT a.worker_number) AS shared_worker_count<br>  FROM zip_dispatch_worker_pairs a<br>  JOIN zip_dispatch_worker_pairs b<br>  ON a.worker_number = b.worker_number AND a.zip_code &lt; b.zip_code<br>  WHERE a.zip_code IN (SELECT zip_code FROM zip_stats)<br>        AND b.zip_code IN (SELECT zip_code FROM zip_stats)<br>  GROUP BY a.zip_code, b.zip_code<br>  HAVING COUNT(DISTINCT a.worker_number) &gt;= 2<br>)<br>SELECT<br>  sw.zip_a,<br>  sw.zip_b,<br>  sw.shared_worker_count * 1.0<br>    / (sa.unique_worker_count + sb.unique_worker_count<br>         - sw.shared_worker_count) AS jaccard_index<br>FROM shared_workers sw<br>JOIN zip_stats sa ON sw.zip_a = sa.zip_code<br>JOIN zip_stats sb ON sw.zip_b = sb.zip_code</pre><p>Two things to notice. The join condition a.zip_code &lt; b.zip_code produces undirected pairs without duplicates, and the zip_stats CTE enforces minimum activity thresholds before the self-join, keeping the edge list sparse.</p><p><strong>Python layer.</strong> Load the edge list from dbt. Filter by the Jaccard threshold. Build a networkx graph with Jaccard as edge weight. Run Louvain community detection (python-louvain or the networkx built-in) with a fixed seed. Louvain is randomized, and without a seed, a re-run on identical data can move zone boundaries. Apply fallback logic for isolated ZIPs. Label each zone by the largest city in the community (HOUSTON_TX, for example). Write the mapping table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rl9_RnVeZt9dqEBNiR9Asg.png" /><figcaption>Commute zone forecasting system architecture — dbt computes the edge list, Python clusters communities, downstream models read zone grain</figcaption></figure><p>The mapping table, commute_zone_mapping, is versioned by effective date. New runs write new rows and old rows stay intact, so historical joins see zones as they were at the time. The largest-city label doubles as a stable zone ID, so the same labor market keeps the same ID across re-derivation runs instead of splitting its history at every version boundary. A companion table, worker_commute_zone_affinity, records each worker’s fractional dispatch share per zone; this is what the supply side reads. Commute patterns change slowly, so quarterly or yearly re-derivation is sufficient. The feature store layer backing these tables is covered in <a href="https://medium.com/p/94c52bab8402">Feature Store Refactoring: Build Once, Reuse Everywhere</a>.</p><h3>Wiring Commute Zones into Demand and Supply</h3><p>Shifts aggregate to commute zone via a point-in-time join to the mapping table. Weekly aggregates replace city-level aggregates at commute zone × job type grain. In the region we ran this on, that took the model from around 400 city-level training units to around 85 zone units. Each zone averaged four to five times more weekly dispatches than its constituent cities had individually. The XGBoost demand model runs at this grain without architecture changes.</p><p>On the supply side, the zone boundary change does something city-level aggregation cannot: it includes workers in the supply signal for locations they haven’t worked yet but plausibly will. A worker who has dispatched across three ZIPs in a commute zone is available labor for all ZIPs in that zone, not just the ones already in their history. In the three-city example from the opening, those fifteen workers now appear in a single zone’s supply signal; the false surplus/shortage split disappears at zone grain. Workers contribute fractionally to each zone weighted by dispatch share, so a worker splitting time between two zones counts toward both.</p><p>With demand and supply on the same grain, the gap is directly comparable at zone level, which drives recruitment prioritization and operational planning.</p><h3>Results</h3><p>The model architecture and feature set were the same in both runs. The geographic unit was the design change.</p><p>At city level, the demand model reached an R² of ~0.60 at a four-week horizon. At commute zone level, R² was ~0.89 at four weeks and ~0.88 at eight. The city baseline was only evaluated at four weeks, so that horizon is the like-for-like comparison.</p><p>Zone-level MAE was ~2.64 workers per week at four weeks and ~2.75 at eight, so error barely grows as the horizon doubles. High week-to-week volatility in temporary staffing makes demand forecasting hard, and R² holding near 0.9 out to eight weeks is strong in this domain. The city baseline of ~0.60 reflects the problem from the first section, where small cities with sparse weekly counts dominate the error.</p><p>Two caveats worth naming. The city and commute zone models were trained and evaluated independently rather than as a controlled ablation, so read the 0.60 → 0.89 jump as an indication of magnitude rather than a precise attribution to geography alone. And these results come from a geography with dense metro structure; rural regions cluster differently, and sensitivity to threshold choice (0.03 vs. 0.05 vs. 0.10) remains an open tuning question.</p><h3>Conclusion</h3><p>Geographic granularity is a modeling decision. The city default is wrong for labor markets where workers routinely cross administrative lines. Commute zones, derived from dispatch history, match how labor markets operate.</p><p>The pipeline is around 300 lines of Python and two dbt models, built once and re-derived on a slow cadence. Every downstream model that touches geography reads the same zone grain, so the investment is shared rather than repeated per model.</p><p>The question I’m investigating next is zone stability over time. If the worker population shifts geographically after a major employer opens or closes in a region, when does re-derivation change the boundaries in a meaningful way? Understanding when existing zones still hold is an open question, and likely worth its own article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f18ec53773b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Feature Engineering: Why Positional Lags Silently Break on Sparse Data]]></title>
            <link>https://medium.com/@piotr.kalanski/time-series-feature-engineering-why-positional-lags-silently-break-on-sparse-data-416c540b07c8?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/416c540b07c8</guid>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-10-08T07:01:02.557Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*miZppqmnL9GA9WfSiIGJxg.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>The density assumption your forecasting pipeline is making, and how calendar-aware transformers fix it</blockquote><p>We were running demand forecasting for staffing placements: hundreds of job categories across a national staffing platform, most with irregular assignment cadences. The pipeline was clean. Unit tests passed. Validation metrics looked exactly where they should be. We pushed to test, watched the first few weeks of predictions roll in, and declared it done.</p><p>A few weeks later, someone noticed the demand forecasts for low-cadence categories were subtly off. Not catastrophically wrong, just quietly and persistently wrong in a way that didn’t show up in aggregate metrics. When we isolated the sparse categories and measured directly, MAPE was roughly 3× higher than our aggregate evaluation suggested, because aggregate MAPE across high-volume categories was swamping the signal from the sparse tail. It took two days of debugging to find the root cause. It wasn’t the model. It wasn’t data drift. It was lag_1 pointing to the wrong week.</p><p>That bug is what this article is about.</p><p>Lag, rolling window, and EWMA features are the foundation of time series ML. They are the first things you reach for, they compose cleanly with sklearn-style pipelines, and they work until they don’t. The failure mode is silent, because it only surfaces on sparse data: series where some time periods have no rows at all. And in demand forecasting for anything with irregular cadence, from staffing assignments to marketplace orders, sparse data is the norm, not the edge case.</p><p>This article covers why positional transformers are wrong for sparse series, what “calendar-aware” means instead, and how to build a pipeline that’s correct by construction, not just on the weekly data in your notebook.</p><h3>Positional Transformers: Conventional Wisdom, and When It Holds</h3><p>Positional transformers are stateless and sort-aware. The property they don’t advertise is that they are also density-dependent, and sparse data exposes that gap.</p><p>The standard toolkit (LagFeatures, RollingWindowFeatures, EWMAFeatures) works by operating on row positions within each group. Every transformer enforces sort_values([group_col, date_col]) before computing anything and carries no fitted state. The density requirement is the part they don’t handle, and they won’t tell you when you’ve left the regime where it matters. This is not unique to one library: the lag and window generators in feature-engine, tsfresh, and sktime are all positional by default, so the density assumption rides along silently no matter which you use.</p><p><strong>LagFeatures</strong> shifts rows within a group by a fixed offset. lag_1 gives you the previous row’s target value, lag_2 gives you two rows back. Fast and correct when the data is dense, meaning every time period has exactly one row.</p><p><strong>RollingWindowFeatures</strong> aggregates over a window of preceding rows. The critical implementation detail is the one-period shift: the window for computing features at period <em>t</em> must exclude period <em>t</em> itself, otherwise you’ve leaked the target. On dense, sorted data, a window of 4 rows is always the same as “the last 4 periods.”</p><p><strong>EWMAFeatures</strong> applies exponential decay, weighting recent observations more heavily. This is where the cracks begin to appear even on nearly-dense data, because the decay rate is calibrated in rows rather than in time units, so a gap of any size is treated identically to no gap at all.</p><p>On a dense sequence, one row per group per week with no missing periods, all three work as intended. Here’s the output for a clean weekly series:</p><pre>week       | target  | lag_1  | rolling_mean_3  | ewma<br> - - - - - | - - - - | - - - -| - - - - - - - - | - - - <br>2024–01–01 | 120     | NaN    | NaN             | NaN<br>2024–01–08 | 134     | 120    | NaN             | 120.0<br>2024–01–15 | 118     | 134    | NaN             | 127.0<br>2024–01–22 | 145     | 118    | 124.0           | 122.5<br>2024–01–29 | 109     | 145    | 132.3           | 133.8</pre><p><em>Parameters: rolling_mean_3 uses window=3, min_periods=3; ewma uses halflife=1, adjust=False; both computed on the one-period-shifted target so the current row is excluded.</em></p><p>Every lag points to exactly the previous week. Every rolling window spans exactly the previous <em>n</em> weeks. Every EWMA decay step represents one week of elapsed time. This is the regime where positional transformers are both correct and efficient.</p><h3>The Core Problem: What Sparse Actually Means</h3><p>Here’s the distinction that matters: a missing period and a zero-value period are not the same thing.</p><p>A week where a job category had zero placements is meaningfully different from a week where that category had no row in the table because it was inactive. In the first case, the model should learn that demand hit zero. In the second case, the model has no information about that week. The absence is the signal, and fabricating a value for it (by implicitly treating it as the prior row) corrupts the feature semantics entirely.</p><p>Sparse series are common wherever activity is event-driven rather than continuous:</p><ul><li>Staffing demand logged only for active assignment weeks</li><li>Demand for long-tail SKUs purchased irregularly</li><li>Event-driven operational metrics recorded only when an event occurs</li><li>Marketplace orders that only exist when a buyer transacts</li></ul><p>For these series, lag_1 does not mean “previous week.” It means “previous row.” And those rows can be one week apart or six months apart, with the transformer having no idea which.</p><p>Here’s what this looks like on a real sparse dataset with two groups at different cadences:</p><pre>group | date       | target | lag_1 (pos.) | rolling_mean_3 (pos.) | actual lag_1 span | actual rolling span<br>------|------------|--------|--------------|-----------------------|-------------------|--------------------<br>A     | 2024-01-08 |   45   |     NaN      |         NaN           | —                 | —<br>A     | 2024-01-22 |   62   |      45      |         NaN           | 2 weeks ✗         | —<br>A     | 2024-03-04 |   38   |      62      |         NaN           | 6 weeks ✗         | —<br>A     | 2024-03-18 |   71   |      38      |        48.3           | 2 weeks ✗         | 10 weeks ✗<br>A     | 2024-04-15 |   55   |      71      |        57.0           | 4 weeks ✗         | 12 weeks ✗<br>B     | 2024-01-15 |   91   |     NaN      |         NaN           | —                 | —<br>B     | 2024-01-22 |   78   |      91      |         NaN           | 1 week ✓          | —<br>B     | 2024-01-29 |   84   |      78      |         NaN           | 1 week ✓          | —<br>B     | 2024-02-05 |   69   |      84      |        84.3           | 1 week ✓          | 3 weeks ✓<br>B     | 2024-02-12 |   73   |      69      |        77.0           | 1 week ✓          | 3 weeks ✓</pre><p>Every Group A lag_1 points further back than one week (two, six, two, then four weeks), while every Group B lag_1 is exactly one week. Positional lag_1 treats Group A’s six-week gap as identical to Group B’s consistent weekly cadence. The rolling window tells the story even more clearly. rolling_mean_3 at row 2024–04–15 for Group A claims to be “the last 3 periods,” but those 3 rows span 12 weeks. Group B’s rolling_mean_3 at 2024–02–12 spans exactly 3 weeks. The model receives features labeled identically that encode completely different temporal windows.</p><p>EWMA has the same problem in a subtler form. Decay counts rows rather than elapsed days, so a 6-week gap gets the same weight as a 1-week gap. The entire temporal structure of the features is wrong, and the model has no way to know.</p><p><strong>The assumption was always there. Sparse data just makes it visible.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y2AhSX2ZBwsJZa3HniwXeg.png" /><figcaption>Positional vs calendar-aware lag: lag_1 on Group A’s sparse series points to a row 6 weeks away under positional semantics, while calendar-aware lag correctly targets the missing Feb 26 slot and returns NaN</figcaption></figure><h3>Calendar-Aware Transformers: A Different Mental Model</h3><p>The conceptual shift is simple: instead of shifting rows by <em>k</em> positions, join on <em>date − k × frequency</em>.</p><p>A calendar-aware lag for lag_1 with weekly frequency doesn’t look at “the row before this one in the group.” It looks up the row whose date is exactly one week before the current one. If that row exists, it returns its value. If it doesn’t exist, because that week had no activity, it returns NaN.</p><p>That NaN is the correct answer. It means “we have no observation for that period,” which is exactly the right signal to pass to the model, because it allows the model to learn the distribution over missing values rather than training on silently fabricated history.</p><h4>The Rule</h4><ul><li><strong>Dense data, confirmed regular cadence</strong> → positional transformers are correct and efficient.</li><li><strong>Sparse data, irregular cadence, or any series where inactive periods exist</strong> → calendar-aware transformers are the default.</li><li>Treat positional lags as an optimization you switch on only after confirming the data is dense, not as the safe starting point.</li></ul><p>The calendar-aware approach also makes cross-group semantics stable. lag_1 on Group A and Group B now means exactly the same thing: “the value one week ago, if it exists.” The temporal meaning is anchored to the calendar, not to the row density of each individual series.</p><h3>Implementation: Calendar-Aware Lag, Rolling, and EWMA</h3><p><em>If you’re already sold on the approach, skip ahead to the Validation section for the pipeline contract and end-to-end assembly.</em></p><p>The implementations below prioritize semantic clarity over performance: exact date joins make the correctness argument self-evident. At scale (tens of thousands of groups, 100+ week histories), merge_asof-based strategies bring down the cost; the semantics carry over unchanged.</p><p>One design choice runs through the lag and rolling transformers: the offset is expressed as day arithmetic (pd.DateOffset(weeks=1)), not as an anchored frequency alias like “W”. pd.tseries.frequencies.to_offset(“W”) is anchored to Sunday, so adding it to a Monday-aligned date snaps forward to the next Sunday instead of adding seven days, and the join then misses every row and returns all-NaN with no error. DateOffset(weeks=1) adds seven days regardless of weekday, so both transformers are anchor-independent by construction. EWMA is the exception, covered in its section.</p><h4>CalendarAwareLagFeatures</h4><p>The join mechanics are the whole point. Periods with no historical row produce NaN naturally, with no masking, no post-hoc correction, and no fabricated values.</p><pre>import pandas as pd<br>from sklearn.base import BaseEstimator, TransformerMixin<br><br>class CalendarAwareLagFeatures(BaseEstimator, TransformerMixin):<br>  &quot;&quot;&quot;<br>  Compute lag features via exact date join rather than row offset.<br>  Returns NaN for periods with no matching historical row.<br>  `period` is day-based (DateOffset(weeks=1)), so it is anchor-independent:<br>  a lag adds exactly k*7 days regardless of which weekday the dates fall on.<br>  &quot;&quot;&quot;<br><br>  def __init__(<br>    self,<br>    lags: list[int],<br>    period=None,<br>    group_col: str = &quot;group&quot;,<br>    date_col: str = &quot;date&quot;,<br>    target_col: str = &quot;target&quot;<br>  ):<br>    self.lags = lags<br>    self.period = period if period is not None else pd.DateOffset(weeks=1)<br>    self.group_col = group_col<br>    self.date_col = date_col<br>    self.target_col = target_col<br>  <br>  def fit(self, X, y=None):<br>    return self # Stateless<br><br>  def transform(self, X: pd.DataFrame) -&gt; pd.DataFrame:<br>    X = X.sort_values([self.group_col, self.date_col])<br>    result = X.copy()<br>    for lag in self.lags:<br>      lookup = X[[self.group_col, self.date_col, self.target_col]].copy()<br>      lookup[self.date_col] = lookup[self.date_col] + lag * self.period<br>      lookup = lookup.rename(columns={self.target_col: f&quot;lag_{lag}&quot;})<br>      result = result.merge(<br>        lookup, on=[self.group_col, self.date_col], how=&quot;left&quot;<br>      )<br>    <br>    return result</pre><h4>CalendarAwareRollingFeatures</h4><p>Each row’s window is the calendar range [date — w*period, date), a stable meaning regardless of data density. The implementation below uses a self-join to make the semantics explicit. It is O(n²) per group, which is intentional for illustrative clarity; for production use at high group counts or long histories, a sorted-merge or merge_asof-based approach brings this down to O(n log n).</p><pre>class CalendarAwareRollingFeatures(BaseEstimator, TransformerMixin):<br>  &quot;&quot;&quot;<br>  Rolling aggregations over calendar windows, not row counts.<br>  Window [date - w*period, date) is consistent regardless of data density.<br>  `period` is day-based (DateOffset(weeks=1)) and anchor-independent.<br>  &quot;&quot;&quot;<br><br>  def __init__(<br>    self,<br>    windows: list[int],<br>    agg: str = &quot;mean&quot;,<br>    min_periods: int = 1,<br>    period=None,<br>    group_col: str = &quot;group&quot;,<br>    date_col: str = &quot;date&quot;,<br>    target_col: str = &quot;target&quot;<br>  ):<br>  self.windows = windows<br>  self.agg = agg<br>  self.min_periods = min_periods<br>  self.period = period if period is not None else pd.DateOffset(weeks=1)<br>  self.group_col = group_col<br>  self.date_col = date_col<br>  self.target_col = target_col<br><br>  def fit(self, X, y=None):<br>    return self<br><br>  def transform(self, X: pd.DataFrame) -&gt; pd.DataFrame:<br>    X = X.sort_values([self.group_col, self.date_col])<br>    result = X.copy()<br><br>    # Self-join on group to pair each row with all potential history rows<br>    hist = X[[self.group_col, self.date_col, self.target_col]].rename(<br>      columns={self.date_col: &quot;hist_date&quot;, self.target_col: &quot;hist_target&quot;}<br>    )<br>    pairs = X[[self.group_col, self.date_col]].merge(hist, on=self.group_col)<br>    for w in self.windows:<br>      col_name = f&quot;rolling_{self.agg}_{w}&quot;<br>      window_start = pairs[self.date_col] - w * self.period<br><br>      in_window = pairs[<br>        (pairs[&quot;hist_date&quot;] &gt;= window_start) &amp;<br>        (pairs[&quot;hist_date&quot;] &lt; pairs[self.date_col])<br>      ]<br><br>      agg_df = (<br>        in_window<br>        .groupby([self.group_col, self.date_col])[&quot;hist_target&quot;]<br>        .agg(count_=&quot;count&quot;, value_=self.agg)<br>        .reset_index()<br>      )<br>      agg_df[col_name] = agg_df[&quot;value_&quot;].where(<br>        agg_df[&quot;count_&quot;] &gt;= self.min_periods<br>      )<br><br>      result = result.merge(<br>        agg_df[[self.group_col, self.date_col, col_name]],<br>        on=[self.group_col, self.date_col],<br>        how=&quot;left&quot;<br>      )<br><br>    return result</pre><h4>CalendarAwareEWMAFeatures</h4><p>Standard EWMA has the deepest problem of the three: decay is applied row-by-row, so a series with a 3-week gap has the same decay as a series with 3 consecutive weeks. The fix is to resample each group to the full dense calendar frequency before computing EWMA, then join back to the sparse original.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qJwBbh410nRRKDNwGLmZlA.png" /><figcaption>Calendar-aware EWMA pipeline: sparse input resampled to dense weekly grid, EWMA applied with ignore_na=False so decay accumulates across empty weeks, then reindexed back to the original sparse rows</figcaption></figure><p>This transformer is the one case where the anchor matters, because resample does need an anchored frequency alias. pandas resample(‘W’) anchors to Sunday by default. If your input dates are Monday-aligned, the resampled index falls on Sundays, and reindex back to the original dates silently returns NaN for every row. Use an anchored alias that matches your data: ‘W-MON’ for Monday-anchored weekly series, ‘W-SUN’ for Sunday. Getting this wrong produces exactly the kind of silent all-NaN output this article is warning against. (The lag and rolling transformers above sidestep this entirely by using day arithmetic instead of resample.)</p><pre>class CalendarAwareEWMAFeatures(BaseEstimator, TransformerMixin):<br>  &quot;&quot;&quot;<br>  EWMA with decay measured in calendar time, not row count.<br>  Resamples each group to dense frequency before applying decay.<br>  freq must match the date anchor of your input data:<br>  - &#39;W-MON&#39; for Monday-anchored weekly series<br>  - &#39;W-SUN&#39; (or &#39;W&#39;) for Sunday-anchored weekly series<br>  Mismatched anchors will produce all-NaN output with no error.  <br>  Note: group loop is straightforward to parallelize with joblib<br>  for pipelines with high group counts (e.g., tens of thousands of SKUs).<br>  &quot;&quot;&quot;<br> <br>  def __init__(<br>    self,<br>    halflife: int,<br>    freq: str = &quot;W-MON&quot;,<br>    group_col: str = &quot;group&quot;,<br>    date_col: str = &quot;date&quot;,<br>    target_col: str = &quot;target&quot;<br>  ):<br>    self.halflife = halflife<br>    self.freq = freq<br>    self.group_col = group_col<br>    self.date_col = date_col<br>    self.target_col = target_col<br><br>  def fit(self, X, y=None):<br>    return self<br><br>  def transform(self, X: pd.DataFrame) -&gt; pd.DataFrame:<br>    result = X.copy()<br>    ewma_records = []<br>    <br>    for group_id, group_df in X.groupby(self.group_col):<br>      group_df = group_df.set_index(self.date_col).sort_index()<br>      <br>      # Resample to dense frequency: NaN for missing periods.<br>      # freq must match the weekday anchor of your input dates.<br>      dense = group_df[[self.target_col]].resample(self.freq).mean()<br>      <br>      # Shift to exclude current period (no leakage)<br>      dense_shifted = dense.shift(1)<br><br>      # ignore_na=False: decay continues across empty periods.<br>      # adjust=False: recursive form, one decay step per resampled week.<br>      ewma = dense_shifted[self.target_col].ewm(<br>        halflife=self.halflife, adjust=False, ignore_na=False<br>      ).mean()<br>      for date, val in ewma.reindex(group_df.index).items():<br>        ewma_records.append((group_id, date, val))<br>        ewma_df = pd.DataFrame(<br>          ewma_records,<br>          columns=[self.group_col, self.date_col, f&quot;ewma_{self.halflife}&quot;]<br>        )<br><br>    return result.merge(<br>        ewma_df, on=[self.group_col, self.date_col], how=&quot;left&quot;<br>    )</pre><p>The ignore_na=False parameter is the critical detail. Without it, EWMA skips missing periods entirely, restoring the positional behavior we’re trying to fix. With it, decay continues accumulating across empty periods, so a halflife of 4 means 4 resampled weeks regardless of whether those weeks had observations. Setting adjust=False uses the recursive form, where each resampled week is one decay step; it is also what produced the EWMA column in the clean-series table above.</p><h3>Validation: Failing Fast Before the Bug Reaches Production</h3><p>Every pipeline stage that operates on (group, date) pairs assumes those pairs are unique. Duplicates don’t fail loudly. They silently corrupt every downstream join. A GroupDateValidator that enforces uniqueness and datetime types at pipeline entry is how you surface these bugs in dev rather than three months into live predictions.</p><pre>class GroupDateValidator(BaseEstimator, TransformerMixin):<br>  def __init__(self, group_col: str = &quot;group&quot;, date_col: str = &quot;date&quot;):<br>    self.group_col = group_col<br>    self.date_col = date_col<br><br>  def fit(self, X, y=None):<br>    return self<br><br>  def transform(self, X: pd.DataFrame) -&gt; pd.DataFrame:<br>    if not pd.api.types.is_datetime64_any_dtype(X[self.date_col]):<br>      raise ValueError(<br>        f&quot;Column &#39;{self.date_col}&#39; must be datetime. &quot;<br>        f&quot;Got {X[self.date_col].dtype}.&quot;<br>      )<br><br>    dupes = X.duplicated(subset=[self.group_col, self.date_col])<br><br>    if dupes.any():<br>      raise ValueError(<br>        f&quot;Found {dupes.sum()} duplicate (group, date) pairs. &quot;<br>        &quot;Downstream features will be incorrect.&quot;<br>      )<br><br>    return X</pre><p>If you need to disable validation for performance in a hot path you’ve already tested, that’s a trade-off you can make deliberately. But make it deliberately. Validation is the contract between your data pipeline and your feature pipeline. Removing it silently is how the lag_1-points-to-the-wrong-week bug gets back in.</p><h4>End-to-End Pipeline</h4><pre>from sklearn.pipeline import Pipeline<br><br>pipeline = Pipeline([<br>  (&quot;validate&quot;, GroupDateValidator(group_col=&quot;group&quot;, date_col=&quot;date&quot;)),<br>  (&quot;lags&quot;, CalendarAwareLagFeatures(lags=[1, 2, 4])),<br>  (&quot;rolling&quot;, CalendarAwareRollingFeatures(windows=[4, 8], agg=&quot;mean&quot;, min_periods=2)),<br>  (&quot;ewma&quot;, CalendarAwareEWMAFeatures(halflife=4, freq=&quot;W-MON&quot;)),<br>])<br>features = pipeline.fit_transform(sparse_demand_df)</pre><p>The lag and rolling stages take no frequency argument: they default to weekly day-arithmetic offsets and work on any weekday alignment. Only the EWMA stage needs freq set to match your date anchor.</p><h3>Closing: Audit Before You Assume</h3><p>Time series features encode assumptions about data density. When those assumptions are implicit, buried in the row-offset mechanics of positional transformers, sparse data breaks them invisibly. The model trains, evaluates, and deploys. It’s just wrong in ways that aggregate metrics don’t catch.</p><p>Positional lags are an optimization for data you’ve already confirmed is dense.</p><p>Before your next model refresh, run this check against your training data:</p><pre># For each group, compute the median gap between consecutive observations<br># and compare it to your declared feature frequency.<br># If they don&#39;t match, you have sparse series, and positional transformers are lying.<br>gaps = (<br>  df.sort_values([group_col, date_col])<br>    .groupby(group_col)[date_col]<br>    .diff()<br>    .dt.days<br>)<br>print(gaps.describe())<br># Expected for weekly data: median ~7 days<br># Median of 14, 21, or higher: sparse series present</pre><p>If the median gap doesn’t match your declared frequency, lag_1 does not mean “previous period” for a meaningful share of your groups. The audit takes five minutes. Finding it three months into production takes two days.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=416c540b07c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Knowledge System Rots Because the Index Is a Lie]]></title>
            <link>https://medium.com/@piotr.kalanski/your-knowledge-system-rots-because-the-index-is-a-lie-17d2430cbd68?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/17d2430cbd68</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[knowledge-management]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-09-04T07:01:02.277Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ft22Vuz9A5MFVGFLxqj7w.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><p>You find the perfect paper in a Claude session, exactly the prior work your experiment needs, and by the next session it’s gone. The conversation ended. The insight never landed anywhere permanent.</p><p>That’s the first failure. The second is worse: you build a system to prevent it. A notes repo, carefully indexed, tagged by research area. Six months later the index is quietly stale, because nobody updated it after the last bulk ingest. You have 40 papers in papers/, 28 entries in index.yaml, and no reliable way to tell which 12 vanished.</p><p>The system that now tracks 330 papers without the index ever drifting didn’t start that way. It started as the same setup described above, and it broke four times before it stopped breaking. The rest of this piece is a deep-dive on the reading subsystem from <a href="https://medium.com/@piotr.kalanski/i-built-a-personal-operating-system-inside-claude-heres-the-architecture-2b2464da1555">I Built a Personal Operating System Inside Claude — Here’s the Architecture</a>.</p><h3>What Broke in Five Versions</h3><p>The design went through five documented revisions before a line of implementation. Each revision closed a specific gap. The same root cause showed up every time: judgment doing a bookkeeping job.</p><p><strong>v1, index drift.</strong> The authored index fell out of sync with the notes it was supposed to describe, because it was treated as authored state rather than generated output. The fix was to derive it instead, and it gets its own section below.</p><p><strong>v2, silent dedup failures.</strong> The same paper got captured twice under different URLs, because two callers normalized those URLs differently. The fix was to centralize identity in a single module.</p><p><strong>v3, area-rename breakage.</strong> Renaming a research area left dangling tags scattered behind it, because the rename was done by hand rather than atomically. The fix was a script that rewrites every reference in one pass.</p><p><strong>v4, silent warnings.</strong> Validation warned correctly, and nobody read the warning. Nothing in the system chose a moment to surface the decay. The fix was a Stop-hook backstop.</p><p><strong>v5, cross-repo writes.</strong> The knowledge repo reached into other repos to keep them current, because integration was push-based. The fix was a strict read-only pull boundary.</p><p>Every fix above is the same move: take a must-never-drift job away from judgment and give it to a deterministic mechanism. The rest of this article unpacks the three load-bearing ones.</p><h3>The Load-Bearing Decision: Index as a Derived Artifact</h3><p>In v1, index.yaml was maintained by whoever was convenient: me, a skill, a script. The result was a cache. Caches drift. They drift because writing to them requires discipline, and discipline is the one resource that runs out.</p><p>The v1 fix was architectural, not disciplinary. The index is now never authored or hand-maintained by anything. It is rebuilt from per-note frontmatter by reindex.py, the same way compiled output is derived from source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j2Aubl04w0hswNUCxTsKgQ.png" /><figcaption>Index as a derived artifact — v1 broken design vs current correct design showing note files flowing through reindex.py into a derived index.yaml</figcaption></figure><p>This matters because of the properties it creates automatically:</p><ul><li>The index can be deleted at any time and rebuilt losslessly from the note files alone.</li><li>It’s committed for convenience (fast reads, diffable history) but is never authoritative.</li><li>The frontmatter in papers/ and books/ is the single source of truth.</li></ul><p>The brittleness in v1 was a rule: “never hand-edit the index.” Rules rot. This converts it into a property: “hand-edit anything you like, the index will be rebuilt.” You cannot violate a property by forgetting it.</p><p>At 330 files, globbing and parsing frontmatter directly is fast enough that the index is an optimization, not a necessity. The system is correct without it. That’s the tell of a correct derived artifact: you could delete it and lose only speed, not correctness.</p><p>Reindex runs in well under one second, a tiny fraction of a Claude session. At current scale it has never been the bottleneck, and there have been zero index-drift incidents since this design went in.</p><h3>Identity and Dedup: Where Personal Knowledge Systems Silently Break</h3><p>The v2 failure looked like this: 12 papers tracked in a research area, with 3 duplicates under different filenames. None flagged. The system thought it had broader coverage than it did.</p><p>The root cause was two callers normalizing URLs differently. One kept the www prefix, one stripped it. One kept the trailing slash, one didn’t. The resulting IDs didn’t collide, so dedup passed. Both entries survived.</p><p>The filename (YYYY-MM-slug.md) is a human convenience, not the identity key. Every note carries a stable id field instead:</p><ul><li>arxiv:2310.03714, with the version suffix stripped, because v1 and v2 of a paper aren’t distinct works</li><li>doi:10.1145/…, lowercased</li><li>isbn:9780262045025, digits only</li><li>url: followed by a SHA-1 of the canonical URL, for everything else</li></ul><p>URL canonicalization lives in exactly one file, ids.py: force https, lowercase the host, strip the www prefix, strip the fragment, strip tracking parameters, strip any trailing slash. Every caller that touches identity imports from it. Validation, the dedup check, and the ingest and discover skills all use the same normalization functions. Two captures of the same paper via different URLs produce the same ID by construction.</p><p>Dedup authority is always the dedup check reading source frontmatter, never the derived index. A stale index can never resurface a paper already tracked.</p><h3>The Three-Layer Taxonomy: Skill, Script, or Hook?</h3><p>With two concrete fixes in place (derived index and centralized identity), the abstraction becomes visible: both fixes are instances of the same move. Mechanical, must-never-drift jobs went to deterministic mechanisms. The question “where does this capability belong?” now has a decision rule.</p><p><strong>Skills own judgment.</strong> Identifying relevant research areas, writing a summary, deciding an open question is worth recording, ranking a discovery shortlist, comparing papers for agreement and gaps. These are fuzzy, contextual, need an LLM. They’re allowed to be imperfect.</p><p><strong>Scripts own bookkeeping.</strong> ID normalization, dedup, index rebuild, schema validation, area-tag rewrites, commits. These are mechanical, integrity-critical, must be 100% reliable. Zero judgment means it’s a script, not a skill.</p><p><strong>Hooks own timing.</strong> When to validate, when to reindex, when to surface a warning, when to refresh statistics. Hooks remove the reliance on remembering to keep things consistent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_TdCemL-HrU6btwinCb7DA.png" /><figcaption>The three-layer taxonomy — Skills own judgment, Scripts own bookkeeping, Hooks own timing, with examples from the reading system</figcaption></figure><p>The pattern test: does this capability need judgment? The deep-dive and compare skills do, so they’re skills. Generating reading statistics doesn’t, so it’s a script. When to generate those statistics is a timing question, so it’s a hook. Same taxonomy, every time.</p><h3>Hooks: The Timing Layer That Removes Discipline as a Requirement</h3><p>The PostToolUse hook fires on every Write, Edit, and MultiEdit, running validation and then reindex after each one. The index is always current. No discipline required. Because the hook lives in this repo’s .claude/ directory, it fires only for edits in this repo. The repo boundary is the scope boundary.</p><p>Validation is warn-not-block. A bad-but-recoverable state stays saveable. This was a deliberate choice over hard-blocking: hard-blocking writes on a validation failure means you can lose work mid-session. Warn-not-block means a bad state persists until it’s fixed, but nothing is lost.</p><p>The problem with warn-not-block is the v4 failure. Warnings nobody reads are the same as no warnings. The answer is the Stop hook.</p><p>The Stop hook re-runs validation at session end and surfaces any outstanding failures as the last thing in the session. It also flags uncommitted changes and papers stuck in reading status for more than 30 days. It reports and never auto-fixes. Fixing takes judgment, which means it’s a follow-up session item.</p><p>There is one cost worth naming. The PostToolUse hook does a full-corpus reindex on every write. Ingesting ten papers fires ten full rebuilds. Sub-second at 330 files, a deliberate “simple until it hurts” choice. The escape hatch is documented: pass only the changed paths to an incremental upsert if full rebuilds ever become slow. Naming this cost is part of the design.</p><h3>Three Post-Design Additions That Prove the Layering Held</h3><p>The design went through five revisions before implementation. These three additions came after, and slotted in without modifying anything structural. They’re the clearest evidence the taxonomy works as a decision rule under change, not just as framing.</p><p><strong>A statistics script run from a hook.</strong> Reading statistics (papers read, per-area counts, coverage gaps) always current, never hand-counted. Pure bookkeeping means script. Timing means hook. Coverage blindness now has a number to show for itself.</p><p><strong>A deep-dive skill.</strong> It generates a summary for a single paper, drilling into reproducibility, methodology, and claimed contributions. Pure judgment, needs an LLM, allowed to be imperfect. Exactly what belongs in a skill, not a script.</p><p><strong>A compare skill.</strong> It compares tracked papers in a research area and surfaces agreement, contradiction, and gaps. This is the synthesis-gap killer: it turns a pile of captured papers into something you can use when writing. Synthesis judgment means skill.</p><p>Each addition took one question, one answer, and one placement decision before a line was written.</p><h3>Integration: How the Reading System Feeds the Rest of the Stack</h3><p>The annotate skill sets an article_seed flag in a note’s frontmatter. The article-brainstorm skill queries the index for that flag as a read-side pull, not a push. The knowledge repo never writes into the articles repo. This article was seeded that way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNcBiZZCRCRms1VtIf-Ixw.png" /><figcaption>The reading workflow — discover via read-discover, capture with read-annotate, synthesize with read-deepdive and read-compare, and pull into the articles repo via a read-only boundary</figcaption></figure><p>Area-map and compare output feeds the outlining skill as research grounding. Papers tagged stack-relevant become cited supporting evidence in architectural decision records.</p><p>Before this system, finding new papers meant roughly an hour per session: talking with Claude to supply context and area, then manually deduplicating against what was already read or queued. Now it’s a single sentence to Claude Code and several new papers land in the queue. The bookkeeping went to a script; the judgment stayed with the skill.</p><p>The cross-repo dependency stays strictly read-only, which is the v5 fix operating at system scale. Integration that pulls can’t create the cross-repo drift that integration that pushes would. The knowledge repo owns its judgment, bookkeeping, and timing internally and exposes only a queryable surface.</p><p>The <a href="https://medium.com/p/20e52a26b605">multi-agent feature workflow</a> I built follows the same Claude Code repo shape: same .claude/ structure, same skill/script/hook taxonomy, applied to a multi-agent development workflow rather than a reading system. This is deliberate. The taxonomy isn’t reading-specific. It’s a general answer to where any capability in a Claude project belongs, and the sibling repo is the proof it transfers.</p><h3>Conclusion</h3><p>The system now tracks 330 papers. The index has never drifted. Drift was made structurally impossible.</p><p>The derived-index decision is the whole game. Get it right and the entire class of two-sources-of-truth bugs disappears. Get it wrong and you’re manually syncing forever, one discipline failure at a time.</p><p>The three-layer taxonomy (judgment to skills, bookkeeping to scripts, timing to hooks) is the generalization. It’s now how I decide where any capability in any Claude project belongs, not just reading systems.</p><p>The v1 failure was a design failure, not a discipline failure. Most personal knowledge systems die the same death. Choosing the right tool for each job is how you stop digging.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17d2430cbd68" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Deck Is the Last Thing I Build: A Claude Code System for Tech Talks]]></title>
            <link>https://medium.com/@piotr.kalanski/the-deck-is-the-last-thing-i-build-a-claude-code-system-for-tech-talks-102a2737a38c?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/102a2737a38c</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[public-speaking]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 21 Aug 2026 07:01:03 GMT</pubDate>
            <atom:updated>2026-08-21T07:01:03.173Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PEXWOOFQGA_6k8dFRphuQw.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>A gated skill pipeline that keeps decks and live demos cheap to change by making sure nothing expensive gets built before the plain text is approved.</blockquote><p>Three slides into a deck, I realized the story needed reordering — and discovered that was now a layout fight, not an edit. Master slides fought placeholders while speaker notes drifted out of sync with the slides they used to describe. What should have taken two minutes ate the rest of the afternoon.</p><p>This is Part 5 of a series applying one pattern (persistent context and custom skills, kept honest by explicit review gates) to concrete, non-code deliverables. <a href="https://medium.com/@piotr.kalanski/i-built-a-personal-operating-system-inside-claude-heres-the-architecture-2b2464da1555">Part 1</a> covered the architecture; this part uses the same idea to prepare internal tech talks: a finished deck plus runnable live demos.</p><p>The payoff below is one gate rule, not a tour of eight skills, and it generalizes to any pipeline where reviewable text is cheap and rendered output is expensive.</p><h3>Why Internal Tech Talks Are More Expensive Than They Look</h3><p>I’ve prepared tech talks repeatedly for two engineering teams, roughly a dozen engineers between them, spanning data platform/ML engineering and data/analytics. The same five failure modes kept showing up, and none of them look expensive until you name the cost.</p><p><strong>Drafting structure inside PowerPoint.</strong> Opening the slide tool first means structure gets negotiated inside the one tool that’s hardest to restructure. Move three slides and you’re fighting layouts, not editing an argument.</p><p><strong>Bullets that only make sense to the person who wrote them.</strong> A bullet reads fine to its author and means nothing to the room, because the real explanation lives only in the speaker’s head, not on the slide.</p><p><strong>Demos reduced to screenshots.</strong> A pasted screenshot or a recorded GIF standing in for a live demo tells the audience one thing: we didn’t trust this to run live. They read that correctly.</p><p><strong>Series amnesia.</strong> In a multi-session deep dive, each session either re-explains material already covered or silently assumes coverage that never happened.</p><p><strong>No rehearsed fallback for live demos.</strong> Live code fails sometimes: a stale credential, a redeployed service. Without a rehearsed fallback, one failure eats the room’s remaining attention for the rest of the talk.</p><p>This is the same move <a href="https://medium.com/@piotr.kalanski/i-stopped-letting-ml-discovery-meetings-pretend-to-be-strategy-this-is-the-workflow-i-use-instead-75e4224df8cc">Part 4</a> made for ML discovery meetings: name the failure modes before naming the system, because the system only makes sense once the cost is visible.</p><h3>The System: Folder Structure and 8 Skills</h3><p>This one lives differently from the rest of the series. Parts 1–4 keep the master-context pattern inside a Claude.ai Project, in project instructions and knowledge files. This is a Claude Code project instead: skills sit under .claude/skills/, auto-discovered by name straight from the repo, versioned in git like anything else in the codebase. Same architecture, different substrate. The pattern doesn’t care which one it runs on.</p><p>The folder structure, one entry per talk series:</p><pre>talks/&lt;series-slug&gt;/<br> _context.md &lt;- speaker voice, audience, series coverage<br> _series-plan.md &lt;- running index, updated after each session<br> session-NN/<br> outline.mdba<br> slide-content.md &lt;- the review gate<br> diagrams/<br> demos/<br> deck.pptx<br> _retrospective.md</pre><p>Eight skills carry a session from a blank page to a finished deck:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VtwJQ0yuAIJRKngAcDK0Nw.png" /><figcaption>Reference table listing all eight skills, what each one produces, and when it runs in the session — from talk-init through the talk-expand gate to the final talk-pptx render</figcaption></figure><p>That’s the scaffolding. The argument is in what step 3 refuses to let step 5 do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHGciXSwiqXacIW_4JqakQ.png" /><figcaption>Session pipeline diagram showing the eight skills from series setup through outline, refine, the slide-content.md review gate, diagrams and demos, to the final PPTX render</figcaption></figure><h3>The One Idea Worth the Whole Article: The Plain-Text Gate</h3><p>deck.pptx is never generated before slide-content.md exists and has been explicitly reviewed and approved in the current session. Ask the system to skip straight to PPTX and it refuses by default. Overriding that refusal takes an explicit instruction, not a shortcut.</p><p>The reason is what each edit type costs once it’s on the other side of PPTX generation:</p><ul><li>Reorder a section → layout reflow across every slide it touches, plus master/placeholder mismatches that need manual repair.</li><li>Split one slide into two → speaker notes desync from the slide they were written to explain.</li><li>Cut a slide → renumbering by hand, and cross-references that now point at nothing.</li><li>Rewrite a bullet → re-touching a slide object instead of editing a line of text.</li></ul><p>In slide-content.md, each of those is a text edit measured in seconds. The whole argument is a cost comparison. Editing a paragraph takes seconds. Repairing a rendered artifact takes a fight with the rendering tool.</p><p>Slide content itself is written as one block per slide: a bullet that’s a prompt for the speaker rather than a complete thought, full speaker notes carrying the real explanation, and inline flags (<em>[DIAGRAM NEEDED], [CODE BLOCK], [TABLE], [DEMO: &lt;label&gt;]</em>) that tell downstream skills what to build.</p><p>The gate isn’t a one-off rule for slides. The same ordering repeats twice more in the pipeline: no slide-content.md before the outline is refined, no demo scaffolding before slide-content.md is approved. Cheap-to-expensive sequencing, enforced structurally rather than left to author discipline.</p><p>I can show what enforcing it actually changed, because I’ve built talks under three different regimes and that turned building custom agent skills into a habit, is what kept me from settling for the middle regime.</p><p>Fully manual prep, writing structure directly in a slide tool with no AI assistance, used to cost days per talk. Moving to AI-assisted prep with an off-the-shelf PPTX-generation skill and a simple outline was already a big jump, but revision still happened by iterating on the rendered deck: several refine passes, each one a Claude Code or Cursor session spent re-touching slide objects one at a time. Slow and token-expensive.</p><p>With the plain-text gate in place, refine iteration on the rendered deck drops to zero or one pass, because the structural churn already happened in slide-content.md, where it was cheap. End to end, that’s days down to hours, plus roughly an hour of slide-review time saved per talk that used to go into reviewing and re-reviewing a rendered deck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JGeCTHUugcV5wUh7F4l1_Q.png" /><figcaption>Comparison of prep time across three regimes: fully manual, AI-assisted iteration on the rendered deck across several passes, and the plain-text gate at zero to one pass</figcaption></figure><p>The middle stage matters more than either end of that comparison. It’s proof I already tried the obvious shortcut, generating the deck early and iterating on the rendered artifact, and hit its cost directly before building the gate that removes it.</p><p>Part 4 made close to the same point about a different kind of premature commitment: forcing a decision instead of preserving options past the point where they’re free. This section makes the equivalent point about edits, forcing the cheap ones to happen before you commit to the expensive artifact.</p><h3>Live Demos as a First-Class Artifact, Not an Afterthought</h3><p>Same gate logic, one tier down. Demos are protected by slide-content.md the same way the deck is.</p><p>talk-demos only runs when slide-content.md contains a [DEMO] flag. Each demo gets its own self-contained folder: a runnable script or notebook, a README covering setup and run instructions plus the talking script, and a fallback artifact that’s non-negotiable, not optional.</p><p>The deck itself never substitutes a screenshot or a GIF for a live demo. It carries a handoff slide pointing at the demo folder, with the return-to-deck moment scripted directly in the speaker notes.</p><p>No live demo has actually failed mid-talk yet. The fallback has never been invoked in practice. I’m not going to manufacture a near-miss to make this section land harder. The honest framing is that it’s deliberate insurance, kept because it’s untested. Building a fallback is cheap. Dead air in front of the room is expensive. The fallback stays a required step, even without a save to point to. If it keeps going unused for long enough, I’ll reconsider requiring it, which is itself the point. The gate answers to a cost-benefit test, not to ritual.</p><h3>Series-Awareness and Audience Calibration</h3><p>_context.md drives two smaller mechanisms, worth a sentence each and secondary to the gate itself.</p><p>For a multi-session deep dive, it’s pasted at the start of every session: speaker voice, audience composition per domain, what the series has already covered, what this session must not repeat. _series-plan.md is the running index talk-series updates after each session. Skip this and a four-session architecture walkthrough degrades into four talks that each re-explain the same diagram.</p><p>The same file also carries the audience default (senior engineers spanning data engineering and ML engineering, plus BI/analytics, addressed at peer register), and lets a single session override it for something like a cross-functional leadership briefing, without touching a single skill.</p><h3>The Deck Is the Last Thing You Should Touch</h3><p>The deck is the last artifact you touch — it’s the most expensive one to change. Every other artifact in this pipeline exists to keep you out of it until the structure is locked.</p><p>The gate generalizes past tech talks. Any workflow with a cheap-to-review text stage and an expensive-to-edit rendered stage benefits from the same structural refusal, refusing to let the tool build the expensive thing until the cheap thing has been reviewed.</p><p>Five domains into this series now: <a href="https://medium.com/@piotr.kalanski/i-built-a-personal-operating-system-inside-claude-heres-the-architecture-2b2464da1555">architecture</a>, <a href="https://medium.com/@piotr.kalanski/the-claude-recruitment-system-that-got-me-interview-ready-in-a-weekend-f9724e51a94d?source=your_stories_outbox---writer_outbox_published---">recruitment</a>, team management, <a href="https://medium.com/@piotr.kalanski/i-stopped-letting-ml-discovery-meetings-pretend-to-be-strategy-this-is-the-workflow-i-use-instead-75e4224df8cc">project planning</a>, and tech talks. The constant isn’t the domain. It’s the gate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=102a2737a38c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Pipeline Spine Pattern: One Base Class That Owns Your ML Lifecycle]]></title>
            <link>https://medium.com/@piotr.kalanski/the-pipeline-spine-pattern-one-base-class-that-owns-your-ml-lifecycle-8c968bec94a1?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c968bec94a1</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-08-07T07:01:02.472Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6oxcijpikSkSEiB4pZjt8Q.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>How a single abstract base class replaced eight diverging training scripts and ran a LightGBM regressor and a reinforcement-learning agent on the same orchestration contract.</blockquote><p>The fourth training script in your codebase is when it stops being a convenience and starts being a problem. The first one worked. The second added MLflow. The third engineer rewrote the data loader their way. The fourth has a temporal-split bug nobody noticed because the tests still pass. You don’t have a training pipeline — you have four slightly different opinions about what a training pipeline is.</p><p>The root cause is structural: no shared contract defines what a training pipeline is and does. Every new model re-decides the same infrastructure questions from scratch (how to load data, how to split safely, how to log experiments, how to tune hyperparameters), and the small differences compound into maintenance debt. A fix in one pipeline doesn’t propagate to the others. A change to MLflow logging means touching every script by hand.</p><p>The fix is one design move: put the orchestration that every pipeline shares into a base class, and expose hook methods where model-specific logic lives. I’ll call it the Pipeline Spine. On a pricing reinforcement-learning research effort I led, the same spine ran a LightGBM regressor and a TD3 RL agent on the same orchestration contract, with different hooks. This article covers both levels (feature engineering and training), what the spine gives you for free, and three design decisions that aren’t obvious but matter in production.</p><h3>Why Monolithic Training Scripts Break</h3><p>A typical monolithic training script handles data loading, splitting, encoding, training, evaluation, and logging in one flat 300–400-line file. At N=1, this is fine. At N=5, it’s a liability.</p><p>Here’s a sanitized version of what the fifth model’s training script often looks like in a codebase without a shared contract:</p><pre># train_demand_model.py - and 7 more files just like it, each subtly different<br>df = load_from_warehouse(&quot;demand_features&quot;) # loader copy #3, slightly tweaked<br>train, test = train_test_split(df, shuffle=True) # random split on time series - leak<br>X = encoder.fit_transform(train) # encoder fit per run - mappings drift<br>model = LGBMRegressor(**hardcoded_params).fit(X, y)<br>mlflow.log_metric(&quot;rmse&quot;, rmse) # whatever this author remembered to log<br>joblib.dump(model, &quot;model.pkl&quot;)</pre><p>Three latent bugs in six lines. The random split on time-series data leaks future signal into training, inflating offline metrics. The encoder fit on each run produces different integer mappings, so a model served from one run can silently mismatch features prepared by another. And the MLflow logging is whatever the author remembered that day, with no consistency across experiments.</p><p>What makes this structural problem hard to see is that pipelines <em>look</em> diverse (LightGBM vs. PyTorch vs. an RL agent) but share the same orchestration sequence. The diversity only appears at a handful of steps: the data transformation logic, the model definition, the evaluation metric. Everything else (loading strategy, splitting, profiling, logging, writing output) is identical across every model you’ll ever build. In a monolithic script, you re-implement that identical infrastructure every time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vd_MuUqHovldQOyw6dXMbw.png" /><figcaption>Before/after comparison: on the left, eight diverging monolithic scripts re-implement the same load-split-encode-train-log sequence, with amber markers on the steps that silently drift; on the right, one Pipeline Spine owns the shared steps in navy and exposes model-specific hooks in teal.</figcaption></figure><h3>The Pattern: What “Pipeline Spine” Means</h3><p>The Pipeline Spine applies two classical patterns to ML pipeline orchestration. Template Method fixes the sequence of lifecycle steps in a base class; subclasses override only the steps that vary by model. Strategy makes infrastructure concerns (data loading, writing) exchangeable implementations selected automatically by environment, without touching calling code.</p><p>The base class is a fixed backbone of lifecycle stages. Each model plugs its specific logic into defined attachment points, the abstract hook methods. The backbone doesn’t change shape; only what hangs from the hooks varies per model.</p><p>This is a thin orchestration shell that uses sklearn, MLflow, and Optuna; it does not replace them. The base class decides when each step happens. The concrete subclass decides what happens at the model-specific steps.</p><p>There are two levels in the system. A feature-engineering spine, <strong>BaseFeaturePipeline</strong>, takes raw data and produces a processed, temporally split dataset. A training spine, <strong>BaseTrainingPipeline</strong>, consumes that dataset and produces a trained, evaluated, logged model. The diagram below maps both lifecycles, showing which steps are base-owned and which are abstract hooks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2wb0jk9ZGFeynTjhDJVFsA.png" /><figcaption>Two-level orchestration: BaseFeaturePipeline (left) and BaseTrainingPipeline (right), each showing base-owned steps in navy, abstract hooks in teal outline, and conditional opt-in steps with a dashed border. An amber connector shows the processed dataset handoff between pipelines.</figcaption></figure><h3>The Feature-Engineering Spine</h3><p>BaseFeaturePipeline owns a fixed ten-step lifecycle: MLflow run setup, feature lookup creation, data loading, raw data transformation, temporal splitting, preprocessing pipeline construction, fit-and-transform, post-processing, output writing, and data profiling. A subclass implements four abstract methods. The base handles the other six steps entirely.</p><pre>class FillRateFeaturePipeline(BaseFeaturePipeline):<br>  <br>  def create_feature_lookups(self) -&gt; list[FeatureLookup]:<br>    # Define which Feature Store entities to join at point-in-time<br>    return [<br>      FeatureLookup(table_name=&quot;worker_features&quot;, lookup_key=&quot;worker_id&quot;),<br>      FeatureLookup(table_name=&quot;shift_features&quot;, lookup_key=&quot;shift_id&quot;),<br>    ]<br><br>  def transform_raw_data(self, df: DataFrame) -&gt; DataFrame:<br>    # Filter rows, apply business rules, derive columns<br>    return df.filter(df.assignment_status.isin([&quot;filled&quot;, &quot;unfilled&quot;]))<br><br>  def create_preprocessing_pipeline(self) -&gt; Pipeline:<br>    # Build the sklearn preprocessing pipeline for this model&#39;s features<br>    return Pipeline([<br>      (&quot;imputer&quot;, SimpleImputer(strategy=&quot;median&quot;)),<br>      (&quot;scaler&quot;, StandardScaler()),<br>    ])<br><br>  def transform_prepared_data(self, df: DataFrame) -&gt; DataFrame:<br>    # Post-split transforms: fill remaining gaps, derived indicators<br>    return df.fillna({&quot;response_time_hours&quot;: df[&quot;response_time_hours&quot;].median()})</pre><p>Everything else (which loading strategy to use, when to split, how to fit the preprocessing pipeline, what to profile, where to write output) the base class handles without any code in the concrete subclass.</p><p>One decision worth calling out: the base class fits the preprocessing pipeline on the training split only, then applies the fitted transformers to val and test. This is the correct behavior, but it’s a decision monolithic scripts routinely get wrong because it requires threading the split boundary through the transform step. When you inherit it from a base class that gets it right once, you stop getting it wrong.</p><p>Feature pipelines built on BaseFeaturePipeline have run Binomial (fill-rate), Poisson (demand), binary classification (customer approval), and multi-target regression (baseline rate) models in production at a large temporary staffing marketplace. Four model families on one orchestration contract.</p><h3>The Training Spine</h3><p>BaseTrainingPipeline extends the backbone with four capabilities the feature spine doesn’t need: training-data sampling, feature selection, hyperparameter tuning, and segment analysis.</p><p>A basic model implements six hooks: get_features_to_load, prepare_data, select_features, train_model, evaluate_model, and on_training_complete. Turning on the built-in Optuna integration adds five opt-in tuning hooks (search space definition, model factory, training function, metric calculator, tuning-data preparation), for eleven total. The tuning infrastructure is available if you want it; you pay for those extra hooks only when you opt in.</p><p>The framework agnosticism comes from one implementation decision: train_model returns Any. The base class never inspects the model object it receives back. That’s what lets the same spine run LightGBM regressors, XGBoost classifiers, PyTorch networks, and RL agents. From the base class’s perspective, train_model is a black box. It receives prepared data and returns a model; what that means is up to the subclass.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8BBJviPVcEmCVwblkk6_pA.png" /><figcaption>Framework agnosticism: prepared data flows into the single train_model() hook, whose return type is Any; four interchangeable implementations (LightGBM regressor, XGBoost classifier, PyTorch network, TD3 RL agent) plug into the same slot, and the base class treats every returned model as an opaque object.</figcaption></figure><p>On a pricing RL research effort I led across two phases (2025 Q1 and 2025 Q4), BaseTrainingPipeline ran 8+ models: fill-rate models at two granularities, a demand model, a customer approval model, a baseline-rate model, and a TD3 RL agent. The LightGBM regressor and the TD3 agent shared one orchestration contract and differed only in train_model. That spine went into a production pilot in 2026 Q2. The design isn’t theoretical — earlier in the same codebase, auditing inherited systems had surfaced temporal-split and encoder-drift bugs that had survived testing undetected; fixing both required touching every affected pipeline by hand. The spine makes that class of problem structurally impossible.</p><h3>What the Spine Gives You for Free</h3><p>The before-state was eight monolithic scripts of 300–400 lines each, each re-implementing loading, splitting, and logging with its own small variations. The after-state is one base class and thin per-model subclasses. The orchestration is written and tested once; each new model contributes only its hooks. Engineers new to the codebase extended the spine for a new model within a week of first seeing it, without hand-holding; the model-specific surface is small enough to learn by reading one existing subclass.</p><p>Six capabilities are worth understanding:</p><h4>Multi-environment support</h4><p>The Strategy pattern handles data loading across five implementations: LocalFileStrategy (CSV/parquet) for local runs, RandomDataStrategy for tests with synthetic data, SparkTableStrategy and SparkQueryStrategy for Databricks, and DatabricksFeatureStoreStrategy for point-in-time feature lookups in production. The strategy is selected automatically from the environment and config parameters. There’s no environment-checking branch (no if env == “local”) in the calling code. The same pipeline runs on a laptop against synthetic data and in production against the feature store. One config change swaps the strategy; no code changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tz2xRsq554rCufYRGH6R3A.png" /><figcaption>Strategy pattern for data loading: five concrete strategies (LocalFile, RandomData, SparkTable, SparkQuery, DatabricksFeatureStore) feed a selector that resolves the right one from environment and config, which the BaseFeaturePipeline then calls through a single interface with no environment branching in its code.</figcaption></figure><h4>Training-data sampling</h4><p>Feature selection and hyperparameter tuning on datasets over one million rows can run for hours. One config parameter (training_data_sample_size=10_000) samples N rows for those two operations only. The final model always trains on full data. Sampling happens after splitting (no leakage), and the sample is seeded for reproducibility. The result is tuning iterations that take minutes instead of hours, which means more trials across wider search spaces and better final configurations.</p><h4>Phase-based hyperparameter tuning</h4><p>The built-in Optuna integration runs an exploration phase (wide search space, many trials) followed by an exploitation phase (narrow space, warm-started from exploration results). Studies persist in SQLite, so an interrupted run resumes where it left off instead of restarting. Studies are automatically named by base name, architecture version, and phase, so experiment tracking stays organized without manual naming discipline.</p><h4>MLflow logging at every step</h4><p>The base class logs at every lifecycle stage automatically: source table and data shape after loading, split sizes with date ranges, feature counts, sampling parameters, tuning trial results, and evaluation metrics. A mid-run crash leaves enough context to diagnose the failure without re-running anything. No code in the concrete pipeline is needed to get this coverage; the hooks execute and the base class handles logging on either side.</p><h4>Data profiling as artifacts</h4><p>Comprehensive data profiling runs before training and persists to MLflow as artifacts. When accuracy drops unexpectedly between model versions, the first question is whether the training data distribution changed; the profiling artifact answers it directly. The comparison becomes a two-line MLflow query rather than a re-run.</p><h4>Automatic segment analysis</h4><p>When evaluate_model returns an EvaluationResults object, the base class runs segment analysis automatically: it logs worst-N performers by any dimensions the pipeline specifies, with no extra code in the concrete subclass. Segment analysis is the step most often skipped on a deadline; making it opt-out rather than opt-in means it happens by default.</p><h3>Three Design Decisions Worth Understanding</h3><p><strong>Temporal split as the default, random as opt-in.</strong> Most ML data in marketplace and platform contexts has temporal dependencies. Today’s demand is correlated with yesterday’s, not next Tuesday’s. Random splitting on time-series data leaks future signal into training, inflating offline metrics and producing production disappointment when real-world performance doesn’t match evaluation. Making temporal split the default inverts the burden: you now have to justify why you’d deviate, instead of remembering to configure it correctly on each new project. Random split requires an explicit opt-in flag. (Temporal leakage is one of several silent failure modes in production ML systems; for the sklearn-layer version of the same class of bug, see <a href="https://medium.com/@piotr.kalanski/7-scikit-learn-pipeline-traps-that-quietly-break-models-in-production-b9411f700ec8">7 Scikit-learn Pipeline Traps That Quietly Break Models in Production</a>.</p><p><strong>Python subclassing over YAML configuration.</strong> A YAML config file can express parameters but not logic. Custom data filters, validation rules, and derived columns all require code logic that YAML cannot express. Python subclassing gives full code expressiveness within a template structure, and each abstract method is independently unit-testable. When you’re debugging a feature pipeline, you can test transform_raw_data in isolation without standing up the full orchestration. The config itself (BaseFeatureConfig and BaseTrainingConfig) is a typed dataclass: serializable, MLflow-loggable in one call, and extensible with a single field declaration per subclass.</p><p><strong>Frozen vocabulary management for categorical encoding.</strong> Fitting a LabelEncoder on training data each run produces different integer mappings across runs. That’s a silent mismatch when you serve a model trained in one run against features prepared by another run. FrozenMultiLabelEncoder loads from a fixed global vocabulary established once and shared across all runs. The base class builds the frozen encoder when a vocabulary is available and falls back to a standard encoder when it isn’t, providing graceful degradation with no boilerplate per pipeline. Encoding drift doesn’t raise an error; the model just serves wrong predictions. During an audit of inherited pricing systems, this turned out to be the harder failure to find: the temporal-split bug inflated offline metrics visibly, but the encoder mismatch only surfaced when tracing why a retrained model disagreed with the serving version at the feature level.</p><h3>Closing</h3><p>The Pipeline Spine concentrates the infrastructure decisions every pipeline shares into one place, so each model contributes only the code that’s genuinely specific to it.</p><p>The code is proprietary; the architecture is not, and that’s the part worth taking. This structure holds for fraud detection, churn prediction, demand forecasting, and RL pricing agents: shared orchestration in the base, model-specific logic in the hooks. The backbone stays fixed; only the hooks change.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c968bec94a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Store Refactoring: Build Once, Reuse Everywhere]]></title>
            <link>https://medium.com/@piotr.kalanski/feature-store-refactoring-build-once-reuse-everywhere-94c52bab8402?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/94c52bab8402</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 07:01:03 GMT</pubDate>
            <atom:updated>2026-07-24T07:01:03.023Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RIr96vg5CTwMFmE3slN5YA.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>How consolidating feature logic into a single dbt backbone cut deployment times from hours to minutes, reduced infrastructure costs by roughly 30%, improved model accuracy by 5%, and laid the foundation for online ML inference</blockquote><p>You’re reviewing a model and something feels off. You dig into the feature definitions, and there it is: the same feature — implemented twice, in two different stacks, with subtly different logic. Which one is the production source of truth? Which one has the bug?</p><p>That moment of uncertainty is how feature logic sprawl announces itself: not with an outage, but as a quiet, compounding tax on every decision your team makes from that point forward.</p><p>This is an account of a structural refactor (not a greenfield rewrite) that moved a feature platform serving 10+ ML models across five use cases from a fragmented multi-stack architecture to a single, dbt-centered backbone on Databricks. The outcomes were concrete: deployment changes that once required close to an hour across two dependent projects now take minutes; AI-assisted SQL generation accuracy jumped from roughly 60% to over 90%; and new location features delivered a 5% improvement in mean absolute error for pricing models. None of these were the explicit goal going in. All of them followed from getting the structure right.</p><h3>The Problem With “It Works, But…”</h3><p>The system worked. Three use cases were in production, ML models were generating value, entity modeling in the warehouse was solid, and early point-in-time patterns were already functioning. This was not a broken system — it was a system approaching the ceiling of its own architecture.</p><p>Feature logic was split across two places: Python-based pipelines that handled some computations outside the warehouse, and dbt models that handled others. Each had evolved independently, shaped by the requirements of the use case that introduced them rather than by any shared contract. The consequences were exactly what you’d expect.</p><p>Duplicate feature definitions appeared across projects. Data types diverged: decimal, double, and oversized bigint columns appeared where float and int would have been correct, exact, and cheaper to compute. Business keys were named inconsistently across tables, meaning every join required a mental translation step. The cross-project dependency structure meant that a single change to shared feature logic required coordinated updates across two separate dbt projects, with orchestration, testing, and deployment happening twice. A careful deployment of one change could consume close to an hour.</p><p>Each issue was manageable in isolation. Together, they created a cognitive load that compounded with every new use case added to the platform. The cost of every change was rising. That trajectory doesn’t flatten on its own.</p><h3>Architecture: One Backbone, Many Use Cases</h3><p>The target architecture replaced fragmentation with four explicit layers: <strong>entities → _core → features → use_cases</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xOIxtENF2cU1ioqr0J63KA.png" /><figcaption>The four-layer dbt backbone — entities, _core, features, and use_cases. Feature logic lives in one place; all use cases inherit updates automatically.</figcaption></figure><p><strong>_core</strong> holds shared transformations, computed once. Anything that multiple feature tables would otherwise recompute lives here. It’s the foundation that prevents duplication from re-emerging.</p><p><strong>features</strong> contains reusable, point-in-time–safe feature tables. This is where business logic lives: not scattered across Python scripts and independent dbt models, but in one place, versioned, tested, and documented.</p><p><strong>use_cases</strong> are deliberately thin. They attach labels to feature snapshots for training and evaluation purposes. No feature logic lives in the use_cases layer. When the logic for a feature changes, one model changes, and every use case that consumes it picks up the update automatically.</p><p>The point-in-time guardrail is enforced with a standard join condition: the fact event timestamp must be strictly less than the spine event timestamp. This is the boundary between a trustworthy feature and a data leakage risk. Applying it consistently across the features layer means it doesn’t need to be re-implemented and re-verified for every use case independently.</p><p>Two standards held the architecture together across all layers: surrogate keys (suffixed *_key) used for feature lookups, clearly separated from business keys; and weekly grain represented as a week_start_date aligned to UTC Monday. Consistent naming and grain across every feature table in the platform.</p><p>This is what “build once, reuse everywhere” actually looks like in practice: multiple ML products sharing a single feature backbone without duplicating logic, re-validating the same transformations, or accumulating independent copies of business rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-92GqXIuQArEOuio4HIFQ.png" /><figcaption>Before and after the refactor: fragmented Python pipelines and two dbt projects replaced by a single four-layer dbt backbone, with measurable improvements across deployment time, compute cost, and AI-assisted SQL accuracy.</figcaption></figure><h3>How the Refactor Was Done</h3><p>The approach was phased and deliberate: centralize shared logic in _core, migrate feature logic into the features layer, strip logic from use_cases, then raise test coverage, documentation, and contracts until the metadata quality supported AI-assisted development. At each phase, behavior stability came first: no advancing until the current phase was validated.</p><p>The goal was to propose the architecture, execute the structural refactor, and hand off to the team to take it to production-grade. Orchestration via Airflow, CI/CD setup, and clustering tuning were taken on by the team as follow-up work. A refactor the team can’t extend and own is a handoff problem deferred.</p><h3>Three Technical Bets That Paid Off</h3><h4>Data Type Discipline</h4><p>The previous state: heavy use of decimal, double, and oversized integer columns throughout the feature tables. The downstream effect was increased memory pressure in Python and Pandas workflows, slower query execution in Databricks, and unpredictable behavior at the edges of type coercion.</p><p>The new standard was explicit: float for continuous features and int for identifiers and counts, with boolean for flags. No exceptions without a documented reason.</p><p>The downstream gain was faster query execution and reduced memory pressure in Pandas workflows. More importantly, getting types right in the batch layer is a prerequisite for clean online serving: mismatched types between batch training and online inference are one of the most common silent failure modes in production ML systems, typically discovered only when a real-time request comes in.</p><h4>Multi-Granularity Location Features</h4><p>The previous architecture aggregated at zone level — roughly 100 geographic zones across the US. That granularity is coarse enough to miss local market dynamics that matter for pricing. Models trained on zone-level features plateau before they should.</p><p>The refactor added ZIP, city, county, and state granularity, covering local bill rates, pay rates, and minimum wages by state. The result was a 5% improvement in mean absolute error for pricing models.</p><p>Adding granularity to the feature layer moved a production pricing model’s accuracy by 5% MAE. The driver was feature granularity, not algorithm changes or additional training data. Feature quality is a model performance lever, not just a platform concern.</p><h4>Entity and Key Standardization</h4><p>Surrogate keys (suffixed *_key) are used for feature lookups. Business keys are kept separate. This distinction is enforced consistently across every table in the platform.</p><p>Inconsistent key semantics compound into inconsistent joins, and inconsistent joins are the first thing that breaks when you try to serve features online. In the batch layer, key standardization is the prerequisite for reliable online serving, not just a modeling preference.</p><h3>Quality as a First-Class Engineering Concern</h3><p>Before the refactor: no column descriptions, sparse dbt tests, no table- or column-level contracts. After: over 90% column documentation coverage (AI-assisted, human-reviewed), hundreds of dbt tests added across the model graph, and contracts in place at both the table and column level.</p><p>The measurable impact on developer productivity was significant. Before the refactor, SQL generated by Cursor had errors roughly 40% of the time — requiring manual iteration or fixes before it could be trusted. After, accuracy exceeded 90%.</p><p>LLMs generating SQL need context: what does this column mean, what type is it, what’s the grain of this table, what keys join to what. When models are well-documented with clear column contracts, the LLM has the context it needs to generate correct SQL on the first attempt. When models lack descriptions and contracts, the LLM is guessing.</p><p>Metadata is infrastructure for AI-assisted development. The ROI compounds as the team grows. Every engineer who joins and uses AI tooling benefits immediately from the documentation that’s already in place. Treating documentation as optional hygiene is a choice to pay that cost repeatedly — in every incorrect SQL generation and every engineering onboarding that starts from scratch.</p><h3>From Hours to Minutes: The Deployment Payoff</h3><p>Before consolidation, a change to shared feature logic required coordinated updates across two separate dbt projects. Dependencies had to be managed manually, testing had to run twice, deployments had to be sequenced. A single careful change could take close to an hour.</p><p>After consolidation, everything lives in one dbt project. Changes are localized. Lineage is clear: dbt lineage shows exactly what depends on what, with no cross-project ambiguity. Deployments take minutes. Infrastructure compute costs came down by roughly 30%, a direct result of eliminating the redundant compute that dual-project orchestration required.</p><p>The secondary effects on team dynamics are equally important. Ownership is unambiguous when there’s only one project to own. Code reviews are simpler, because reviewers need to understand one set of conventions rather than two, and onboarding is faster because the whole platform team shares a single vocabulary for features in a single place. The architecture made the cost of every change predictable and low. That predictability is what lets teams move fast without accumulating fear.</p><p>The infrastructure cost reduction this contributed to is covered in more depth in <a href="https://medium.com/@piotr.kalanski/how-we-cut-ml-infrastructure-costs-by-50-without-touching-the-models-c711115cfa59">How We Cut ML Infrastructure Costs by 50% Without Touching the Models</a>.</p><h3>The Bridge to Online Inference: Why dbt Is Not the Bottleneck</h3><p>The common objection: “dbt is a batch tool. How does this help with real-time features?”</p><p>The objection misidentifies the bottleneck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*luWbeCaX2h2DyHhHjGap2g.png" /><figcaption>How the batch feature backbone bridges to online inference: the same entities, keys, and types built for batch training serve real-time predictions directly — no parallel feature engineering required.</figcaption></figure><p>On Databricks as a data lakehouse, data hits the bronze layer as streaming tables. dbt models running on top of that can be refreshed multiple times per day. Views can serve near-real-time reads without full recomputes. Latency is an orchestration and architecture problem, not a dbt constraint, and both are solvable.</p><p>The real bridge to online serving is entity consistency. Same entities, business keys, and data types across batch training and online inference. When those are consistent by design, enforced at the _core and features layer, serving features online doesn’t require a separate feature engineering effort. The data modeling work is already done. What remains is a serving layer problem: routing a request, looking up a key, returning a value.</p><p>The first use case moving to real-time pricing inference is built directly on top of the batch feature backbone. No parallel feature engineering effort, no separate set of business rules to maintain — the batch architecture extended cleanly into online serving because it was built with consistent entities and types from the start.</p><h3>What Follows From Getting the Structure Right</h3><p>Feature platforms fail not because the logic is wrong, but because it lives in too many places. Every additional copy of a feature definition is a liability: something that can drift, diverge, or contradict. Consolidation isn’t cleanliness — it’s what makes every subsequent change cheaper.</p><p>None of these outcomes were the stated goal of the refactor. The 5% MAE improvement, the jump in AI-assisted SQL accuracy, the hours-to-minutes deployments — all followed from two structural decisions: where feature logic lives, and how well it is described. Get those right, and the rest compounds.</p><p>As real-time ML inference becomes the default for production ML products, the teams that invested in clean batch feature architecture will move faster. The work of entity design, key standardization, and type discipline only needs to happen once. The teams that skipped it will be doing it anyway, just under more pressure, with less time to do it right.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94c52bab8402" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your Claude Code Workflow Is Still a Junior Dev (And What I Built to Fix It)]]></title>
            <link>https://medium.com/@piotr.kalanski/why-your-claude-code-workflow-is-still-a-junior-dev-and-what-i-built-to-fix-it-20e52a26b605?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/20e52a26b605</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-07-17T07:01:02.835Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWTlJNRQVbIitZHS3rpAxQ.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>A multi-agent feature pipeline that turns Claude Code from a tool you steer into one that manages itself</blockquote><p>You’ve delegated the code. You’re still the tech lead.</p><p>That’s the ceiling most Claude Code users hit. The code coming out is good, often better than anything you’d write alone. But the coordination is entirely yours: prompt for the plan, review the plan, prompt for implementation, prompt for a code review, apply the fixes, run the tests, notice that docs weren’t updated, commit. Each step becomes a separate Claude session. Each session starts cold.</p><p>I counted my sessions on a recent feature and landed at seven. Seven cold-context restarts, each requiring me to re-establish what we were building, where we were in the process, what had been reviewed versus applied. The code was good. I was doing all the management.</p><p>After 2000+ commits with AI-assisted coding as my primary workflow, I started feeling the ceiling of the one-step-at-a-time model. So I built something different. You provide a draft feature spec, the workflow handles the rest, and you make two decisions during the run. The output is a branch with a clean commit history and a PR ready for your final review.</p><p>Here’s the architecture, the constraint that shapes it, and what I learned building it.</p><h3>The Junior Dev Model (and Its Ceiling)</h3><p>The pattern has a name. You’re the tech lead, Claude is the junior developer. You assign tasks, review output, unblock, assign the next task. This works. The code quality is high. But shipping a feature end-to-end requires you to manually sequence across six or seven separate sessions: explore the codebase, write a plan, review it if the design feels uncertain, implement, run a code review, apply the fixes, run tests, update docs, commit, push. Every session starts cold.</p><p>The overhead isn’t any individual step. It’s the switching cost. Between sessions, you hold the mental model: what was decided in the plan, what was reviewed versus applied, whether the tests covered the edge case you flagged. That’s management work, not engineering work. You’ve delegated execution but kept coordination.</p><p>That raises the obvious question. What would it look like to delegate the coordination itself to Claude Code: the sequencing, stage handoffs, quality gates, and state tracking?</p><h3>The Constraint That Shapes Everything</h3><p>The first instinct when building a multi-agent Claude Code system is to create an orchestrator subagent that spawns planners, implementers, and reviewers in sequence. That doesn’t work.</p><p>Claude Code subagents cannot spawn other subagents. As of mid 2026, this is a documented platform limitation. If you don’t know this going in, you waste days building an architecture that won’t run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NXg8l7GCWk73EdwzkTe03A.png" /><figcaption>Two-panel diagram comparing the rejected subagent-spawns-subagent pattern with the valid skill-in-main-session pattern.</figcaption></figure><p>The limitation narrows the design to one valid structure. The orchestrator must run in the main session as a <a href="https://medium.com/@piotr.kalanski/i-built-a-personal-operating-system-inside-claude-heres-the-architecture-2b2464da1555">skill</a>. The skill spawns subagents via the Agent tool. Subagents never spawn anything. The pattern is <em>skill → N subagents</em>, never <em>subagent → subagent</em>.</p><p>This turns out to be the right shape anyway. The main session is where human gates live, where state is held, where decisions get made. Subagents are pure workers: given a task, produce an artifact, exit. Keeping orchestration in the main session makes the human-in-the-loop design cleaner, not more constrained. The platform limitation forced the better architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*jWavpNd8dkjX45HS7qimqw.png" /><figcaption>The multi-agent feature workflow pipeline — from feature spec to push/PR, showing all five agents, two human gates, and the Small mode bypass.</figcaption></figure><h3>The Pipeline: Five Agents, Two Modes</h3><p>Five specialized subagents, each with a specific model and a constrained tool allowlist:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4C0bwSs4o0uOkwCQHDoV2A.png" /></figure><p>Opus runs on the two roles requiring judgment: planning and reviewing. Sonnet handles execution. Read-only constraints on the planner and reviewer are deliberate. An agent that can’t write code can’t corrupt your codebase while thinking.</p><p>Two modes govern which subagents run:</p><p><strong>Standard</strong> runs the full pipeline: planner → human approval gate → implementer → script review → LLM reviewer → human triage gate → refactorer → doc-updater → final tests → push/PR. Use this for new features, refactors, or anything with design surface area.</p><p><strong>Small</strong> skips the planner. For bug fixes and one-liners, the planning step is overhead. You pay for Opus only where you need it.</p><p>The workflow takes a draft feature specification you’ve prepared as input. The spec goes through the planner before any code is written. What you type to invoke it:</p><pre>/feature-workflow area: auth scope: agent/auth/ &quot;Add JWT expiry validation&quot;</pre><p><em>area</em> sets the git branch prefix, producing a branch named <em>feature/auth-jwt-expiry-&lt;sha7&gt;</em>. The SHA suffix disambiguates repeat runs on the same feature. <em>scope</em> constrains the implementer to specific paths. The quoted string is the full feature spec. That’s the entire invocation for standard mode.</p><p>Agents coordinate through a shared YAML state file (<em>tmp/workflow-state.yaml</em>). Each agent owns a single block: <em>impl:, review:, refactor:</em>, and so on. No agent may overwrite a sibling’s block. The implementer reads and writes only <em>impl:</em>; the reviewer reads <em>impl:</em> but writes only <em>review:</em>. If an agent needs to be retried, only its block goes stale; other agents’ work is preserved.</p><h3>Human Gates (Only Where They Actually Matter)</h3><p>Two human decision points in the entire run.</p><p>Before the first line of code, the workflow captures baseline test failures. Pre-existing failures don’t count as regressions. Only new failures relative to that baseline will block the push at the end.</p><p>Gate 1 comes in standard mode after the planner finishes. You read the plan doc, edit if needed, then reply <em>proceed</em> or <em>revise: &lt;feedback&gt;</em>. Feedback loops back to the planner. You approve the plan before a single line of code is written. The plan is where ambiguous specs become concrete design decisions. Catching a misinterpretation here costs nothing. Catching it in review costs the full implementation.</p><p>Gate 2 comes after LLM review. BLOCKING findings are auto-applied without discussion. RECOMMENDED and OPTIONAL findings go to a triage prompt with a strict grammar: <em>all-recommended, none, F2 F5, all-recommended except F3</em>. Unrecognized input is rejected and re-prompted.</p><p>The alternative is accepting free-text triage input and parsing intent. Vague input produces inconsistent state: findings get partially applied, the state file diverges from reality, and the next agent reads corrupted context. The triage grammar is small enough to memorize after one run.</p><p>Everything else runs without interruption. Pre-flight, commits at defined checkpoints, tests, doc updates, push — no prompts, no confirmation. The workflow surfaces you only when it has a decision to offer.</p><h3>What I Actually Learned Building This</h3><h4>Order quality gates by cost, not by thoroughness</h4><p>Ruff and mypy run before the LLM reviewer. They catch mechanical issues for near-zero cost. Spending Opus tokens reviewing code that would fail ruff is waste. The same static checks run again after refactoring, catching regressions the refactorer introduces. Both passes use the same script; the second pass is the one v1 didn’t have.</p><h4>Warn-only scope discipline is more honest than auto-revert</h4><p>The first version auto-reverted out-of-scope file changes. Auto-revert was the wrong policy. It’s both BLOCKING and unresolvable; you can’t code your way out of a revert. Warn-only is the right approach. The implementer records what it touched and why, the reviewer flags it as a scope-creep finding, and the human decides.</p><h4>Per-agent memory turns repeated runs into accumulated knowledge</h4><p>Three agents maintain a gitignored <em>MEMORY.md</em> across runs: planner, implementer, and reviewer. Codebase-specific lessons accumulate: “every new scorer forgets the <em>eval_strategy: out_of_scope</em> branch,” “auth changes always need a migration test.” The refactorer has no memory — its job is mechanical application of pre-approved findings, and accumulated judgment would be a liability there.</p><h4>The gap you miss in design is almost always a handoff</h4><p>Two additions came after running v1: static analysis with ruff and mypy, and the doc-updater subagent. Both were obvious in hindsight. Documentation was consistently missed because it fell outside the main implementation loop — between the implementer finishing and the reviewer starting, no agent owned it. Adding a dedicated subagent made it non-optional. The same pattern held for static analysis: the gap became obvious after watching the LLM reviewer spend tokens on code that would fail ruff. Both gaps were handoffs, not steps. Design tends to get the steps right and miss the seams between them.</p><h3>What Changes When You Delegate Coordination</h3><p>Delegating coordination means encoding the management work once and staying out of the way except at the two decisions that need a human. Seven sessions for that complex feature, two or three otherwise, all requiring constant steering prompts between steps. Now it’s one run with two brief stops.</p><p>The platform constraint is the load-bearing fact: subagents can’t spawn subagents. Understand it first and the rest of the architecture follows. Start with small mode and a single subagent before adding the full pipeline.</p><p>The tooling will change. Agent Teams will mature, platform limits will lift. But the platform forced something worth keeping: orchestration in the main session, execution in subagents, human in the loop only where a decision is real. That separation is the shape worth building around — whether or not the limit ever lifts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20e52a26b605" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Taught Cursor Our ML Conventions — With 8 Markdown Files]]></title>
            <link>https://medium.com/@piotr.kalanski/how-we-taught-cursor-our-ml-conventions-with-8-markdown-files-c6174da1c8cc?source=rss-4504af9f33e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6174da1c8cc</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Piotr Kalanski]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-07-10T07:01:02.178Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_5uuKh34X53qi7cepH_raw.png" /><figcaption>Generated by author using ChatGPT</figcaption></figure><blockquote>Agent Skills encode your team’s conventions so the AI gets it right the first time, every time.</blockquote><p>You ask Cursor to add a dependency. It writes <em>pip install lightgbm</em>. You’re using <em>uv</em>. You fix it. Tomorrow, same session, same mistake — <em>pip install</em> again. You ask it to define a model config. It returns a plain Python dict. Your team uses dataclasses. You correct it. It puts new code in the wrong module. You correct that too.</p><p>This is the hidden tax of AI coding assistants nobody talks about: not that they can’t code, but that they code for <em>a generic repo that doesn’t exist</em>. They don’t know your conventions or the decisions your team made six months ago. Every correction is time you’re not building, and the AI will make the same mistake in the next session.</p><p>We hit this wall hard. Over the past three quarters, we’ve been building production ML systems almost exclusively through Cursor and/or Claude Code, enough sessions to map every recurring failure pattern. We started tracking corrections: how much time was the team spending fixing AI output versus shipping work? We kept a simple tally in our weekly retrospectives over six weeks. The pattern was clear: in a typical session we were making 10–15 corrections that had nothing to do with logic or business rules. Just conventions. After shipping the first three Skills, that dropped to 3–4. The AI wasn’t failing at code. It was failing at <em>context</em>. So we built a system to fix that.</p><p>We call them Agent Skills: eight Markdown files that encode our team’s entire ML workflow. Prompts got shorter because we stopped re-explaining conventions every session. Generated code fit our architecture on the first attempt. This article walks through what they are and how you can build your own.</p><h3>The Problem Is Context, Not Capability</h3><p>The corrections we kept making weren’t random. They clustered around the same handful of issues: wrong package manager, wrong config pattern, wrong module, wrong inheritance chain. The AI wasn’t confused about Python. It was missing the institutional knowledge that every engineer on our team carries implicitly.</p><p>The fix isn’t a smarter model. Every major model makes the same category of mistakes when your codebase deviates from whatever “standard” their training data assumed. The fix is structured context: a way to tell the AI, at the moment it needs to know, exactly how your team does things.</p><p>Every correction you make is a signal. It means something your team knows hasn’t been encoded anywhere the AI can read. Skills are how you close that gap.</p><h3>What a Skill Is — and How It Works</h3><p>A Skill is a Markdown file stored in <em>.cursor/skills/</em> in your repo. It has two parts: a YAML frontmatter block with <em>name</em> and <em>description</em> fields, and a Markdown body with operational guidelines and a Common Mistakes table.</p><p>The <em>description</em> field is the trigger. It’s not a command you type; the AI reads all your Skill descriptions automatically and decides which ones are relevant based on your prompt. Ask it to add a dependency, and it loads the <em>uv</em> Skill. Ask it to define a training pipeline, and it loads <em>ml-model-training</em>. The decision is semantic, not syntactic.</p><pre>---<br>name: uv<br>description: &gt;<br> Use this skill for any Python package management task, running Python scripts,<br> or setting up virtual environments. Triggers when the user wants to install,<br> add, or remove packages, or run any Python command.<br>---<br><br>## Guidelines<br>- Always use `uv add &lt;package&gt;` - never `pip install`<br>- Run scripts with `uv run python script.py`<br>- Initialize projects with `uv init`<br><br>## Common Mistakes<br>| Mistake | Correct |<br>| - - - - -| - - - - -|<br>| `pip install lightgbm` | `uv add lightgbm` |<br>| `python train.py` | `uv run python train.py` |<br>| `pip freeze &gt; requirements.txt` | `uv pip freeze &gt; requirements.txt` |</pre><p>Skills are version-controlled and shared across the team. The moment someone adds a rule, everyone benefits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_o-LDxtkFZ6n4O7dg-Lm9Q.png" /><figcaption>How Agent Skills work: from developer prompt to convention-compliant code via semantic matching and skill loading</figcaption></figure><p>Skills also compose. A single prompt can fire multiple Skills at once. When I ask the AI to build a Feature Store model, it loads the <em>feature-engineering-dbt</em> Skill and the <em>using-dbt-for-analytics-engineering</em> Skill together. The Feature Store Skill even delegates SQL generation to the dbt Skill.</p><h3><strong>Skill Design Principles That Actually Work</strong></h3><p>The quality of a Skill determines whether the AI follows it reliably. Four principles matter most.</p><p><strong>Be prescriptive, not descriptive.</strong> A Skill that says “we use dataclasses for configuration” is weaker than one that says “always define configs as <a href="http://twitter.com/dataclass"><em>@dataclass</em></a> classes, not plain dicts unless the keys are dynamic at runtime.” Tell the AI what to do, not what exists.</p><p><strong>The Common Mistakes table is your highest-ROI section.</strong> A two-column table of wrong versus correct patterns is the fastest way to shape AI behavior. Show the AI the exact mistake to avoid, not the rule behind it. Every time you catch a recurring error, it belongs in this table.</p><p><strong>Push depth into <em>references/ </em>folders.</strong> The main Skill file should be scannable. For Skills with complex rules (our ML training Skill, for instance), we use a <em>references/</em> subfolder with one file per framework. The AI loads only the relevant reference based on what you’re building. This keeps the main file tight without sacrificing coverage.</p><p><strong>Anticipate the AI’s own rationalizations.</strong> For complex, dependency-heavy workflows, the AI will invent reasons to skip steps — “this change looks simple,” “the downstream impact is minimal.” A Skill can include a “Rationalizations to Resist” table listing shortcuts the AI might argue for, paired with the correct response. Senior engineers apply this discipline automatically; the Skill makes it explicit for the AI. When the AI argues for a shortcut, the Skill tells it to complete the step anyway.</p><h3>Our 8 Skills: The Full ML Workflow</h3><p>Here’s what we built, organized by category. For each Skill, the design choice matters as much as the content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3lvrNW72hAijo6AIxlJIAw.png" /><figcaption>The 8 Skills organized by workflow category: Dev Environment, ML Training, Feature Engineering Python, Feature Engineering dbt delegation chain, and Platform</figcaption></figure><h4>Dev Environment: uv and dataclasses</h4><p>These two Skills cover the fundamentals that affect every file in the repo.</p><p>The <em>uv</em> Skill enforces our package manager unconditionally: <em>uv add</em>, not <em>pip install</em>; <em>uv run</em>, not <em>python</em>. The Common Mistakes table has caught these in every new session since we shipped it.</p><p>The <em>dataclasses</em> Skill is equally simple: prefer <em>dataclass</em> for structured data and configs; use plain dicts only when keys are dynamic at runtime. The AI used to default to dict-based configs, since that’s what most training data looks like. Now it generates dataclass configs correctly, every time.</p><h4>ML Training: ml-model-training</h4><p>This is our most architecturally significant Skill. It encodes a layered base class system we built for our ML framework.</p><p>The core insight: developers should implement only the parts that vary. For LightGBM and XGBoost pipelines, that’s two methods: <em>prepare_data</em> and <em>on_training_complete</em>. Everything else is pre-built in the base class: MLflow setup, train/validation splitting, feature selection, hyperparameter tuning, model training, evaluation, and SHAP explanations. For PyTorch, you add <em>create_trainer</em>.</p><p>The Skill has a <em>references/</em> folder with one file per framework: the LightGBM reference includes default hyperparameter search spaces, the PyTorch reference covers six model architecture types and embedding dimension guidelines, and the XGBoost reference documents its key behavioral differences from LightGBM. When you ask the AI to create a new pipeline, it loads the right reference and generates code that fits our exact inheritance chain: the right class names, method signatures, and config fields.</p><p>The result is framework-compliant pipeline classes on the first attempt, instead of standalone scripts with dict configs and no MLflow.</p><h4>Feature Engineering — Python: feature-engineering-python</h4><p>Feature engineering in our stack follows the same base class pattern as training. You extend <em>FeaturePipelineBase</em> and implement four methods. All transformations must go through an sklearn <em>Pipeline</em>. This isn’t stylistic; it ensures the same code runs at training and inference. Silent skew between training and serving is one of the most common production ML bugs — I wrote about <a href="https://medium.com/@piotr.kalanski/7-scikit-learn-pipeline-traps-that-quietly-break-models-in-production-b9411f700ec8">seven specific ways it fails in production</a> separately.</p><p>The Skill also knows our existing transformers: <em>RelativeFeatureTransformer</em> for ratio features, <em>CyclicFeatureEncoder</em> for time-of-day and day-of-week features, <em>MultiValueLabelEncoder</em> for multi-value categoricals. It reuses these instead of reinventing them, and scaffolds new pipeline classes with correct boilerplate.</p><h4>Feature Engineering — dbt: A Three-Skill Chain</h4><p>Our dbt coverage is split across three Skills that form a delegation chain: validation → analytics engineering → CLI execution. We split it because each layer has a different trigger and a different failure mode; combining them into one Skill made the descriptions too broad to trigger reliably.</p><p><strong>feature-engineering-dbt</strong> owns Feature Store validation. Is the business key correct? Is the timestamp column in UTC? Is point-in-time logic using strict <em>&lt;</em> instead of &lt;=? Does the model avoid PII? This Skill delegates SQL work to the next one in the chain.</p><p><strong>using-dbt-for-analytics-engineering</strong> covers the broader dbt workflow. Its most important feature is the Iron Rule: before touching any table, complete all six discovery steps: understand the source, trace the lineage, check existing tests, review downstream consumers, confirm the grain, and verify the naming convention. This prevents the AI from “helpfully” rewriting a model without understanding its dependencies. The Skill puts principle four into practice with a “Rationalizations to Resist” table, and it has caught reasoning errors mid-generation that would have shipped wrong models.</p><p><strong>running-dbt-commands</strong> enforces <em>uv run dbt — select</em> across every dbt interaction. The <em>— select</em> flag prevents accidental full-project builds that scan every table in the warehouse; on a large project, that’s expensive and disruptive.</p><p>On the validation side: the Feature Store checklist has caught PIT join bugs before they reached production. A PIT join using &lt;= instead of &lt; introduces future data leakage — the model trains on information it wouldn’t have had at prediction time. That bug is silent. The model trains, evaluates, deploys, and only reveals itself through degraded live performance. The checklist stops it at code review.</p><p><strong>Platform: databricks</strong></p><p>The Databricks Skill covers CLI usage and SQL execution against Unity Catalog. When building new features, I instruct the AI to query Unity Catalog metadata directly (schema and column types) before writing any transformation code. What used to be a manual step (open the Catalog, inspect the schema, switch back to the editor) now happens inside the same Cursor session, against what actually exists. It’s a small shift, but it eliminates an entire class of bugs where generated code references columns that don’t exist or have different types than assumed.</p><h3>How to Build Your First Skill</h3><p>Several paths, same format.</p><p>Cursor has a built-in “Create New Skill” command that scaffolds the YAML frontmatter and body. Anthropic published an <a href="https://github.com/anthropics/skills/tree/main/skills/skill-creator">open-source meta-Skill on GitHub</a> for generating new Skills: describe what you want to encode, and it writes the Skill file. Or you copy an existing Skill and edit it. Copy-and-modify is fast.</p><p>The harder question is: where do you start?</p><p>Start with the Skill that would eliminate your most frequent correction. Frequency drives ROI, not complexity. If you correct the AI for the same mistake three times a week, that’s your first Skill.</p><p>Write the Common Mistakes table first. The mistakes are concrete; the guidelines follow from them. Once you have the table, the <em>description</em> trigger is usually obvious: it’s the category of task that causes the mistakes.</p><p>Push it to the team repo immediately. Don’t polish it before sharing. The value of a Skill compounds across the whole team from day one. A rough Skill that eliminates one correction for five engineers is worth more than a perfect Skill still sitting in a draft branch.</p><h3>Conclusion</h3><p>Every time you correct an AI coding assistant, you’re looking at a Skill you haven’t written yet. The correction itself is the spec.</p><p>That correction-rate drop, from 10–15 to 3–4 per session, is the leading indicator. The lagging one is the class of production defects that never ship: the PIT join that leaks future data, the feature pipeline that silently diverges between training and serving.</p><p>The compounding effect isn’t any individual correction saved: your codebase becomes a place where the AI’s institutional knowledge grows alongside yours. Each Skill added raises the floor for every engineer on the team, including the ones who join next quarter. The AI stops being a generic assistant that happens to be pointed at your repo, and starts behaving like a contributor who knows how you work. That kind of leverage doesn’t come from a smarter model. It comes from better context, and better context is something you can build.</p><p><em>What’s the first Skill you’d write for your ML stack? Drop it in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6174da1c8cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>