<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stas Sajin on Medium]]></title>
        <description><![CDATA[Stories by Stas Sajin on Medium]]></description>
        <link>https://medium.com/@stassajin?source=rss-d9a8a91ee988------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_lhtVf3-sKwIkNo0lPEATw.png</url>
            <title>Stories by Stas Sajin on Medium</title>
            <link>https://medium.com/@stassajin?source=rss-d9a8a91ee988------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:13:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stassajin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Are we all p-hacking?]]></title>
            <link>https://stassajin.medium.com/are-we-all-p-hacking-ce002be000d3?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce002be000d3</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 03:15:34 GMT</pubDate>
            <atom:updated>2025-11-10T03:15:34.478Z</atom:updated>
            <content:encoded><![CDATA[<p>If you spend enough time around academics, you hear the same story a hundred different ways: p-hacking is real, it’s old, and it’s a problem they have been talking about with something between guilt and resignation for a few decades. There are <a href="https://journals.sagepub.com/doi/pdf/10.1177/0956797611417632">papers</a> about it, <a href="https://datacolada.org/">careers</a> built on diagnosing it, <a href="https://metascience2021.org/">conferences</a> where people argue about how to stop it.</p><p>P-hacking is the default behavior of smart people under pressure to keep winning. Nobody wakes up planning to corrupt the evidence; they wake up planning to get promoted, to justify headcount, to keep a project alive. Under those incentives, every analytic “judgment call” tilts the same direction, and the system gradually fills up with results that are statistically “significant”</p><p>When I was building an experimentation platform in a previous role, I got to watch this problem up close. My personal KPI was simple: I wanted a tight correlation between the company’s growth and the number of ideas we tried. If revenue was growing 30% year over year, I wanted to see something just as loud in the logs: 30% more experiments, 30% more “wins,” 30% more features marching into production, 30% more adoption in WAU of the internal experimentation platform. If the business was on fire and the experiment graph was flat, something was broken.</p><p>What I actually saw was not “no impact,” but a kind of quietly embarrassing mismatch. The business was growing, say 30% year over year on the metrics that mattered: revenue, contribution margin, the stuff that shows up in board decks. At the same time, our experimentation metrics were exploding. The number of experiments run, the number of teams using the platform, the count of “wins” we logged, those were doubling, sometimes more. In internal dashboards, we were becoming an experimentation machine, with hundreds of experiments launched a month; in the P&amp;L, we were… just a solidly growing company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4pSmrgNa3JYAGcIrkmzOfQ.png" /><figcaption>One line whispers “steady progress”; the other screams “revolution.”</figcaption></figure><p>Staring at those two lines, the obvious question was: where did all that extra “impact” in the experiments go?</p><h3><strong>More knobs, more paths, more ways to be wrong</strong></h3><p>Here’s the uncomfortable part: getting smarter at experimentation doesn’t necessarily make you safer. In a lot of practical settings, it makes you better at p-hacking. If you don’t know much statistics and you’re stuck with whatever the experimentation platform gives you, your degrees of freedom are limited by the UI. But once you know your way around the machinery, the option space explodes. You know you can flip CUPED on or off, change the lookback window, include or drop certain covariates, switch between ratio and difference, log-transform the metric, slice to “high-intent” users, or quietly retire a bad geo. None of those knobs are evil; most of them exist to make analysis better. The problem is that more knowledge translates directly into more choices, and under asymmetric incentives, where “interesting” results are rewarded and boring ones are ignored, those choices don’t average out. They lean toward whatever makes the result look like a win.</p><p>The corollary is that some of your genuinely best analysts are also the ones most capable of p-hacking. They’re the people who, after running a clean top-line read, instinctively ask “where does this work?” and “where doesn’t it?” They slice by device, cohort, country, tenure, intent, traffic source. They try different variance reductions, different windows, different metric definitions. That curiosity is exactly what makes them good, and exactly what gives them a much larger<a href="https://sites.stat.columbia.edu/gelman/research/unpublished/p_hacking.pdf"> garden of forking paths</a> to get lost in.</p><p>My conclusion is that more statistics and education is not the solution if you want more reliable outcomes from experimentation.</p><h3>Pre-registration is the solution</h3><p>If p-hacking is what happens when we make too many choices after seeing the data, pre-registration is just the boring act of locking in a few of those choices before we start. It doesn’t work because people suddenly become virtuous; it works because it shrinks the space of moves you’re allowed to make once the results are on your screen. You can still explore, poke at the data, and slice it twelve different ways — that’s how you generate ideas. But for the part that’s going to drive a decision, you have to commit in advance to what question you’re asking and how you’ll decide if the answer is “yes” or “no.”</p><p>There’s at least one very striking natural experiment that says pre-registration matters a lot. <a href="https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0132382">Kaplan and Irvin</a> looked at 55 large, publicly funded cardiovascular trials supported by the US National Heart, Lung, and Blood Institute between 1970 and 2012. Before 2000, when registration on ClinicalTrials.gov wasn’t yet standard, 17 of 30 trials (57%) reported a significant benefit on their primary outcome. After 2000, when all of these big trials were prospectively registered, only 2 of 25 trials (8<strong>%)</strong> did. The drugs didn’t suddenly stop working in 2000; what changed was that investigators now had to declare their primary outcome in advance. When the authors looked at other outcomes in those post-2000 preregistered trials, they found plenty of significant effects that could have been promoted to “the main result” if nobody had forced them to pre-commit. Almost half of those later trials could have been spun as positive if they’d been allowed to choose the most flattering endpoint after the fact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*gvu4PJQMG7utGtyr.png" /></figure><p>When it comes to the result you’re going to hang a roadmap decision or a promotion packet on, the point of pre-registration is that you’ve already put your chips on the table: this is the hypothesis, this is the metric, this is the window, these are the subgroups, here’s when we stop. Everything else goes in the “exploratory” bucket, clearly labeled as such. You don’t need a ClinicalTrials.gov for product teams to get most of the benefit. A lightweight, time-stamped analysis plan in your experimentation tool, a norm that says “we judge you on whether you followed the plan, not whether you got a win,” and a culture that treats clean nulls as real contributions will take you most of the way there. It doesn’t make p-hacking impossible, but it does make it obvious and changes how people behave.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce002be000d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to run CUPED at scale]]></title>
            <link>https://stassajin.medium.com/how-to-run-cuped-at-scale-d6b2f8aa8bf0?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6b2f8aa8bf0</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Mon, 25 Aug 2025 07:20:43 GMT</pubDate>
            <atom:updated>2025-08-25T13:03:38.702Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve worked on many iterations of the CUPED method and from a practical standpoint found that CUPED is one of those ideas that looks simple on a slide and gets tricky in production. In practice, you’re juggling messy behavior and at scale you’re also juggling a huge cloud bill. For large companies that run thousands of concurrent experiments implementing naive version of CUPED can be super expensive (easily in the tens of millions of dollars in compute costs). This post is a field guide for getting more signal per dollar from CUPED that routinely deliver <strong>100×–1000×</strong> efficiency in the right conditions without giving up precision. We’ll focus on four tactics:</p><ul><li>picking the right pre-experiment window</li><li>caching pre-experiment covariates</li><li>using delta method with CUPED</li><li>sampling when we compute CUPED aggregates</li></ul><p>And a quick note on why this matters. I know I mentioned above compute efficiency and cloud bills. But the real constraint isn’t just money, it’s time. When pipelines are expensive, teams ration freshness due to budget constrains: “we only refresh daily or hourly,” “no realtime monitoring,” “kick off a job and check back in 30 minutes.” That slows learning and turns an experiment platform into a reporting tool. The playbook below is about cutting variance and latency so answers arrive while the question is still interesting.</p><p>I’ll start with the most visible and most often mis-specified choice: the window.</p><h3>Selecting a pre-experiment window for CUPED</h3><p>When people say “use pre-experiment data as a covariate,” it hides a real question: which pre-experiment data? If your experiment runs 7 days and you have two weeks of history, you’ve got several options. The choice matters for both variance reduction and compute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsuxcGPm8FHuv22ry3s6wg.png" /></figure><h4><strong>Option 1 — Match the experiment window (week −1 only)</strong></h4><p><em>Use 7 days of history for a 7-day experiment. </em>This is the default most teams start with (see Fig. A). It’s simple and respects recency, but it often underperforms in consumer settings because pre-period data are sparse. If many users don’t transact every week, week −1 has a lot of zeros, which limits how much variance you can soak up. If you have more data available about your users in your history, why not use more data?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*fTe32qim7Cn205o1xI9JIQ.png" /><figcaption>Figure A</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*3k5OZOXBfmaeCEfXfxq2Bg.png" /><figcaption>Most of your users are going to be like user 2 that don’t transact every week</figcaption></figure><h4>Option 2 — A longer single covariate (weeks −2 &amp; −1 summed)</h4><p><em>Use a 14-day total as one covariate</em> (Fig. B). By pooling two weeks, you reduce zeros and get a stronger proxy for latent “propensity to spend.” That almost always improves variance reduction relative to Option 1. The downside is that you have to process twice the amount of pre-experimental data, which doubles your compute costs and has users waiting for longer to see results refreshing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*TeSWxy24HpbfgkqaojIz9g.png" /><figcaption>Fig B</figcaption></figure><h4>Option 3 — Two non-overlapping covariates (week −1 and week −2)</h4><p><em>Use both weeks as separate covariates</em> (Fig. C). This typically beats Option 2 because the regression can learn different weights for each week. If week −1 is more predictive (recency), it gets a bigger weight; if week −1 had a promo/holiday, the model can down-weight it and lean on week −2. Either way, residual variance drops, so the SE on the treatment effect tightens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*Ukowv7nIIoub7yU_xEqw5g.png" /><figcaption>Fig C</figcaption></figure><h4>Option 4 — Overlapping pair (week −1 and weeks −2 &amp;−1)</h4><p>This is not a strictly better option than option 3, but it is something to watch out for because it tends to increase compute requirements. In this scenario we use two covariates: one spans week -1 and one spans week -1 &amp; -2. If you do that, you will generally get the same estimate as in option 3 but you have to reprocess overlapping data multiple times. Overall, I discourage using this default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*UVDPtRhBu_5h66UUcSUI4Q.png" /><figcaption>Fig 4</figcaption></figure><h4>Some results</h4><p>Here are some OLS results that demonstrate how these methods would looks like. Note that I’m using data generation process that on purpose mimics having users that don’t contribute purchases for the week before experiment but do have data for weeks further back. Overall in practice Option 3 will get the best value out of CUPED.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*fL0dt0zSvV29rOj3eAUoTg.png" /></figure><h3>Caching pre-experiment user level aggregates</h3><p>The advice above suggests that you should leverage more historical data and more covariates to get best variance reduction from CUPED. The downside of that advice is that you will have a very large increase in compute requirements. Nonetheless, the best thing about CUPED inputs is also the simplest: for a fixed anchor date, they never change. Once you know a user’s GMV in week −1 and week −2 relative to an experiment start, those numbers are deterministic (ignoring late arrivals). Recomputing them for every analysis is just lighting money on fire.</p><p>Why this matters? If you have user telemetry, you’ll find that most folks that run experiments are super impatient. A 7-day experiment often gets re-run 100+ times — scheduled jobs, ad hoc refreshes, curious PMs. If you don’t cache randomization-unit aggregates (user/store/courier, etc.), every single of those 100+ runs recomputes identical pre-experiment data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*Tmy1P3w8zBUUCrjeyXTzhA.png" /><figcaption>Example sowing that once a user enters your experiment, pre-exp covariates can just be computed once and cached.</figcaption></figure><p>There’s another dividend to this caching: users are shared across experiments. A single user can sit in hundreds of concurrent tests, which means the same cached aggregates can be reused across pipelines. This trick alone can drive 100x improvement in pipeline efficiency.</p><h3><strong>Delta Method + CUPED</strong></h3><p>For years, the most expensive part of experiment analysis wasn’t the math it was the commute + data movements. Data lived in the warehouse; analysis lived on a big EC2 box. The loop looked like this:</p><ul><li>pull a giant table out of the warehouse,</li><li>ship it across the wire,</li><li>run a regression to get estimates and standard errors.</li></ul><p>Steps 1 and 2 dominate the bill. The Delta method flips this around. Instead of hauling rows to Python, you turn rows into a handful of aggregates you can compute right in the warehouse: means, variances, covariances (metric with covariates and covariate with covariates). This is why in recent years Statsig/Eppo has adopted delta method as the default solution for their metric computation with a rebranding called <a href="https://www.statsig.com/warehouse">warehouse-native</a>. There are quite a few articles describing delta method <a href="https://tech.instacart.com/unlocking-66-faster-experiments-with-covariate-adjustment-power-analysis-for-clustered-data-3a0c11c7baa6">here</a>, <a href="https://srome.github.io/CUPED-with-Multiple-Covariates-and-A-Simpler-Delta-Method-Calculation/">here</a> and <a href="https://medium.com/@rachel.hwung/how-to-use-delta-method-to-estimate-uncertainty-c2b3436a34f6">here</a> and I’m not going to cover the theoretical underpinnings. But the general highlight is that delta method shifts 99% of compute to the warehouse.</p><h4>What you need to compute (warehouse-side inputs)</h4><p>Here’s the birds-eye view of the pipeline for a metric like total gmv per randomization unit. We take the exposures table to define who’s in the test and when they started, roll up event data into a per-user metric (e.g., total GMV and its denominator) over the experiment window, and join a cached tile of pre-period covariates. Then we collapse everything to a tiny set of moments per arm (and once pooled): counts, means, variances, and covariances. That’s the whole warehouse-native pipeline. No user-level rows leave the warehouse, and the app layer just consumes this small bundle of &lt;1kb of aggregates to run delta method through.</p><pre>-- PARAMETERS<br>--   :exp_id<br>--   :treatment_window_days    -- e.g., 7<br>--   :anchor_date              -- DATE used to select the pre-period covariate tile<br><br>WITH exposures AS (<br>  -- One row per user in the experiment (deduped to first exposure)<br>  SELECT<br>    e.user_id,<br>    e.arm,                                    -- &#39;control&#39; or &#39;treatment&#39;<br>    MIN(e.first_exposure_time) AS t0<br>  FROM experiment_exposures e                 -- (experiment_id, user_id, arm, first_exposure_time)<br>  WHERE e.experiment_id = :exp_id<br>  GROUP BY 1,2<br>),<br><br>metric_events AS (<br>  -- Event-level facts for the metric window; join to exposures to prune by t0<br>  SELECT<br>    ev.user_id,<br>    ev.event_ts,<br>    ev.gmv_numerator_atom     AS gmv_num_atom,   -- e.g., GMV dollars<br>    ev.gmv_denominator_atom   AS gmv_den_atom    -- e.g., active_day flag; use 1 if not a true ratio<br>  FROM events ev                                 -- (user_id, event_ts, gmv_numerator_atom, gmv_denominator_atom)<br>),<br><br>metric_per_user AS (<br>  -- Aggregate metric numerator/denominator per user, anchored on t0<br>  SELECT<br>    ex.user_id,<br>    ex.arm,<br>    /* GMV numerator: sum over [t0, t0 + D) */<br>    COALESCE(SUM(CASE<br>      WHEN me.event_ts &gt;= ex.t0<br>       AND me.event_ts &lt;  DATEADD(&#39;day&#39;, :treatment_window_days, ex.t0)<br>      THEN me.gmv_num_atom ELSE 0 END), 0.0)::FLOAT AS gmv_numerator,<br><br>    /* GMV denominator: sum over [t0, t0 + D) (set to 1.0 if your metric is a mean) */<br>    COALESCE(SUM(CASE<br>      WHEN me.event_ts &gt;= ex.t0<br>       AND me.event_ts &lt;  DATEADD(&#39;day&#39;, :treatment_window_days, ex.t0)<br>      THEN me.gmv_den_atom ELSE 0 END), 0.0)::FLOAT AS gmv_denominator<br>  FROM exposures ex<br>  LEFT JOIN metric_events me<br>    ON me.user_id = ex.user_id<br>  GROUP BY 1,2<br>),<br><br>covariate_tile AS (<br>  -- Cached pre-period covariates keyed by (user_id, anchor_date).<br>  -- Add as many *_numerator / *_denominator pairs as you need.<br>  SELECT<br>    ct.user_id,<br>    /* Covariate: pre-period GMV in days -7..-1 */<br>    COALESCE(ct.pre_gmv_1_7_numerator,   0.0)::FLOAT AS pre_gmv_1_7_numerator,<br>    COALESCE(ct.pre_gmv_1_7_denominator, 1.0)::FLOAT AS pre_gmv_1_7_denominator,<br>    /* Covariate: pre-period GMV in days -14..-8 */<br>    COALESCE(ct.pre_gmv_8_14_numerator,   0.0)::FLOAT AS pre_gmv_8_14_numerator,<br>    COALESCE(ct.pre_gmv_8_14_denominator, 1.0)::FLOAT AS pre_gmv_8_14_denominator<br>  FROM user_covariates ct   -- (user_id, anchor_date, pre_gmv_1_7_numerator, ..., pre_gmv_8_14_denominator, …)<br>  WHERE ct.anchor_date = :anchor_date<br>),<br><br>per_user AS (<br>  -- One row per user with metric + covariates<br>  SELECT<br>    m.user_id,<br>    m.arm,<br>    m.gmv_numerator,<br>    m.gmv_denominator,<br>    COALESCE(c.pre_gmv_1_7_numerator,   0.0) AS pre_gmv_1_7_numerator,<br>    COALESCE(c.pre_gmv_1_7_denominator, 1.0) AS pre_gmv_1_7_denominator,<br>    COALESCE(c.pre_gmv_8_14_numerator,   0.0) AS pre_gmv_8_14_numerator,<br>    COALESCE(c.pre_gmv_8_14_denominator, 1.0) AS pre_gmv_8_14_denominator<br>  FROM metric_per_user m<br>  LEFT JOIN covariate_tile c<br>    ON c.user_id = m.user_id<br>),<br><br>-- Sample means/variances/covariances per arm<br>arm AS (<br>  SELECT<br>    arm,<br>    COUNT(*)                                   AS n,<br><br>    -- Metric aggregates (GMV numerator/denominator)<br>    AVG(gmv_numerator)                         AS mu_gmv_numerator,<br>    AVG(gmv_denominator)                       AS mu_gmv_denominator,<br>    VAR_SAMP(gmv_numerator)                    AS var_gmv_numerator,<br>    VAR_SAMP(gmv_denominator)                  AS var_gmv_denominator,<br>    COVAR_SAMP(gmv_numerator, gmv_denominator) AS cov_gmv_num_gmv_den,<br><br>    -- Covariate aggregates: pre_gmv_1_7 (numerator/denominator)<br>    AVG(pre_gmv_1_7_numerator)                 AS mu_pre_1_7_num,<br>    AVG(pre_gmv_1_7_denominator)               AS mu_pre_1_7_den,<br>    VAR_SAMP(pre_gmv_1_7_numerator)            AS var_pre_1_7_num,<br>    VAR_SAMP(pre_gmv_1_7_denominator)          AS var_pre_1_7_den,<br>    COVAR_SAMP(pre_gmv_1_7_numerator,<br>               pre_gmv_1_7_denominator)        AS cov_pre_1_7_num_den,<br><br>    -- Covariate aggregates: pre_gmv_8_14 (numerator/denominator)<br>    AVG(pre_gmv_8_14_numerator)                AS mu_pre_8_14_num,<br>    AVG(pre_gmv_8_14_denominator)              AS mu_pre_8_14_den,<br>    VAR_SAMP(pre_gmv_8_14_numerator)           AS var_pre_8_14_num,<br>    VAR_SAMP(pre_gmv_8_14_denominator)         AS var_pre_8_14_den,<br>    COVAR_SAMP(pre_gmv_8_14_numerator,<br>               pre_gmv_8_14_denominator)       AS cov_pre_8_14_num_den,<br><br>    -- Cross-aggregates: metric ↔ covariate components<br>    COVAR_SAMP(gmv_numerator,   pre_gmv_1_7_numerator)    AS cov_gmv_num__pre_1_7_num,<br>    COVAR_SAMP(gmv_numerator,   pre_gmv_1_7_denominator)  AS cov_gmv_num__pre_1_7_den,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_1_7_numerator)    AS cov_gmv_den__pre_1_7_num,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_1_7_denominator)  AS cov_gmv_den__pre_1_7_den,<br><br>    COVAR_SAMP(gmv_numerator,   pre_gmv_8_14_numerator)   AS cov_gmv_num__pre_8_14_num,<br>    COVAR_SAMP(gmv_numerator,   pre_gmv_8_14_denominator) AS cov_gmv_num__pre_8_14_den,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_8_14_numerator)   AS cov_gmv_den__pre_8_14_num,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_8_14_denominator) AS cov_gmv_den__pre_8_14_den<br><br>  FROM per_user<br>  GROUP BY arm<br>),<br><br>-- Pooled (control + treatment) for CUPED weights β<br>pooled AS (<br>  SELECT<br>    &#39;pooled&#39;                                   AS arm,<br>    COUNT(*)                                   AS n,<br><br>    AVG(gmv_numerator)                         AS mu_gmv_numerator,<br>    AVG(gmv_denominator)                       AS mu_gmv_denominator,<br>    VAR_SAMP(gmv_numerator)                    AS var_gmv_numerator,<br>    VAR_SAMP(gmv_denominator)                  AS var_gmv_denominator,<br>    COVAR_SAMP(gmv_numerator, gmv_denominator) AS cov_gmv_num_gmv_den,<br><br>    AVG(pre_gmv_1_7_numerator)                 AS mu_pre_1_7_num,<br>    AVG(pre_gmv_1_7_denominator)               AS mu_pre_1_7_den,<br>    VAR_SAMP(pre_gmv_1_7_numerator)            AS var_pre_1_7_num,<br>    VAR_SAMP(pre_gmv_1_7_denominator)          AS var_pre_1_7_den,<br>    COVAR_SAMP(pre_gmv_1_7_numerator,<br>               pre_gmv_1_7_denominator)        AS cov_pre_1_7_num_den,<br><br>    AVG(pre_gmv_8_14_numerator)                AS mu_pre_8_14_num,<br>    AVG(pre_gmv_8_14_denominator)              AS mu_pre_8_14_den,<br>    VAR_SAMP(pre_gmv_8_14_numerator)           AS var_pre_8_14_num,<br>    VAR_SAMP(pre_gmv_8_14_denominator)         AS var_pre_8_14_den,<br>    COVAR_SAMP(pre_gmv_8_14_numerator,<br>               pre_gmv_8_14_denominator)       AS cov_pre_8_14_num_den,<br><br>    COVAR_SAMP(gmv_numerator,   pre_gmv_1_7_numerator)    AS cov_gmv_num__pre_1_7_num,<br>    COVAR_SAMP(gmv_numerator,   pre_gmv_1_7_denominator)  AS cov_gmv_num__pre_1_7_den,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_1_7_numerator)    AS cov_gmv_den__pre_1_7_num,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_1_7_denominator)  AS cov_gmv_den__pre_1_7_den,<br><br>    COVAR_SAMP(gmv_numerator,   pre_gmv_8_14_numerator)   AS cov_gmv_num__pre_8_14_num,<br>    COVAR_SAMP(gmv_numerator,   pre_gmv_8_14_denominator) AS cov_gmv_num__pre_8_14_den,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_8_14_numerator)   AS cov_gmv_den__pre_8_14_num,<br>    COVAR_SAMP(gmv_denominator, pre_gmv_8_14_denominator) AS cov_gmv_den__pre_8_14_den<br>  FROM per_user<br>)<br><br>SELECT * FROM arm<br>UNION ALL<br>SELECT * FROM pooled;</pre><h3>Subsampling the covariate aggregates</h3><p>Even after you cache pre-period features and move CUPED into the warehouse, there’s one more easy win: estimate the CUPED weights on a sample, then apply them to the full experiment.</p><p>CUPED needs one thing learned from data: a small set of weights (one per covariate) that tell you how much to subtract from the metric. Those weights come from simple aggregates (means, variances, covariances). But do you really need ALL users to get those estimates? Let’s imagine you have 1M users in your experiment, do you really have to compute CUPED inputs by examining weeks of history for all of them? The answer is no.</p><p><strong>Why this works?</strong></p><ul><li><strong>Aggregates stabilize fast: </strong>We’re not fitting a complex model; we’re estimating a handful of averages and correlations. If your experiment has 1M users, sampling 20% to compute CUPED aggregates for historical data would still be huge.</li><li><strong>The estimate stays honest</strong>: Note that the sampling is random and focused on the unit of randomization. This will keep the CUPED effect unbiased.</li><li><strong>Precision is basically the same</strong>: The final statistics end up being very similar to the original where we use the full sample. You can expect 2–3% variation relative to the original if you deal with aggregates like count distinct. Note that aggregates like min/max will not work though.</li><li><strong>Cost drops a lot</strong>: Because we’re removing the pricey part of computing user level aggregates for multiple covariates, the cost drops by a lot.</li></ul><p><strong>Example</strong>: here are some results of what subsampling impact has on the data generation process. In here I use the delta method and made a small change to compute CUPED inputs on the warehouse on a much smaller sample of users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*EWTtq02A7eqxx7OMNilY_Q.png" /></figure><p>You can see that the ATE and SE are very close to the model that has to compute CUPED and scan ALL the data.</p><h3><strong>Conclusion: The real win is time</strong></h3><p>Cost matters, but what compounds is speed, how quickly a question turns into an answer, and an answer into the next question. Slow pipelines create a curiosity tax; people stop poking at outliers, wait for the nightly job, and make blunter decisions. Use the four tactics above and that tax disappears. Results show up while the ramp is still live, PMs iterate the same day, analysts chase the interesting insights, and engineers ship smaller, safer ramps because they can see what is happening in real time. Overall, if you want your experimentation pipelines to scale efficiently and not have users waiting, you must leverage the options above.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6b2f8aa8bf0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When should data teams worry about platform costs? ]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://stassajin.medium.com/when-should-data-teams-worry-about-platform-costs-4713f0ce1c1d?source=rss-d9a8a91ee988------2"><img src="https://cdn-images-1.medium.com/max/1760/0*baZRmsxyoz9XDWe5" width="1760"></a></p><p class="medium-feed-snippet">I generally find that the way data grows and the way we price its storage and computation follow entirely different trajectories relative&#x2026;</p><p class="medium-feed-link"><a href="https://stassajin.medium.com/when-should-data-teams-worry-about-platform-costs-4713f0ce1c1d?source=rss-d9a8a91ee988------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://stassajin.medium.com/when-should-data-teams-worry-about-platform-costs-4713f0ce1c1d?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/4713f0ce1c1d</guid>
            <category><![CDATA[data-infrastructure]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Mon, 30 Dec 2024 19:06:59 GMT</pubDate>
            <atom:updated>2025-09-18T19:22:39.314Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How should we allocate headcount?]]></title>
            <link>https://stassajin.medium.com/how-should-we-allocate-headcount-af3400ef3fc8?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/af3400ef3fc8</guid>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[strategy]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Mon, 25 Nov 2024 17:15:06 GMT</pubDate>
            <atom:updated>2024-11-25T17:17:28.758Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CnY5Xmy1rvj6igwG" /><figcaption>By <a href="https://unsplash.com/photos/person-pointing-white-paper-on-wall-7tXA8xwe4W4">mariogogh</a></figcaption></figure><p>There’s a story about an early Amazon meeting where Jeff Bezos pulled out a chair and left it conspicuously empty. “This chair,” he told the room, “is for the customer.” The gesture was simple, but the message was clear: every decision, no matter how small, should be made with the customer in mind. Over time, that empty chair became a metaphor for how Amazon thought about prioritization — if it didn’t serve the customer, it didn’t belong in the room.</p><p>But in most companies, it’s not an empty chair that defines prioritization — it’s a spreadsheet. At first, those rows and columns feel harmless, tools for tracking users, revenue, and churn. But there’s a moment in every tech company’s lifecycle when the spreadsheet stops being a tool for reflection and starts driving the conversation. What began as a way to measure the business becomes the scaffolding for decisions about its future: where to invest, what to build, and most critically, where to put people.</p><p>Deciding where to allocate headcount is deciding where to allocate <strong>potential</strong>. It’s deciding what the company is betting on, what it’s letting go of, and what it’s willing to risk. Done poorly, headcount planning is the fastest way to kill innovation and morale. And yet, for something so critical, most headcount allocation processes are shockingly obscure and undefined.</p><p>This post will reflect my opinions on how to optimally allocate headcount. Note that the focus will be on staffing teams and not projects, since team allocations are harder to change once a commitment has been made.</p><h3>The challenges</h3><p>Headcount allocation is uniquely challenging because it’s not just a numbers game — it’s a political one. Every team believes their work is the most important, and they often have the data to back it up. Product teams come armed with ambitious roadmaps and revenue projections. Support teams point to mounting ticket backlogs that are driving down customer satisfaction. Platform or infrastructure teams wave metrics about tech debt and system fragility that could bring the company to its knees. And because resources are finite, these conversations can feel like zero-sum games, where one team’s gain is another’s loss.</p><p>The problem is compounded by how these priorities are presented. Teams with visible issues — customer complaints, missed deadlines, or a feature slipping on the roadmap — fan the flames to make their problems seem urgent. A delayed launch becomes a fire, and fires demand immediate attention. Meanwhile, quieter but equally important work — like automating internal processes, scaling infrastructure, or exploring new product ideas — doesn’t scream for resources. It’s harder to argue for preventing problems that haven’t happened yet, even when the long-term costs of neglecting them could be catastrophic.</p><p>This dynamic reminded me of a conversation I once had with a former manager about headcount allocation feeling reactive, even counterproductive. Teams, I argued, would fan the flames of their fires, ignoring internal inefficiencies, preventative measures, or lack of prioritizing team velocity, because headcount allocation seemed to reward creating problems in need of solving. At some extreme, headcount planning can lead to the resources going to the least productive teams that don’t have a track record of executing well, but which have a lot of fires.</p><p>She nodded and shared an anecdote of her own. “It’s like having two kids,” she said. “If one is special needs, you can’t help but focus more on the struggling child. Even if you know deep down that fairness is important, it’s hard not to be a parent and let your attention and energy go where the cries are the loudest.” She explained that for managers, the instinct to support struggling teams — even at the expense of others — is natural, but it doesn’t always lead to the best outcomes for the whole. And it’s that instinct — well-intentioned but reactive — that often drives bad allocation decisions.</p><h3>The Allure of Simple Answers</h3><p>The problem with headcount allocation is that everyone wants it to be simple. There’s a deep temptation to treat it like balancing a household budget: a fixed pot of resources, a list of needs, and a calculator to sort it all out. For many companies, this instinct hardens into process. Rows of last year’s headcount get copied into this year’s spreadsheet, adjusted by a few percentage points here and there, and approved with little fanfare. It’s efficient, predictable, and deeply flawed.</p><p>Take <strong>historical allocation</strong>, one of the most common approaches. It assumes that whatever a team had last year is a reasonable proxy for what it needs this year. In some cases, this works well enough — teams with consistent, ongoing work can maintain stability. But as priorities shift, historical allocation becomes an anchor. A team that once mattered deeply, perhaps driving early growth, continues to soak up resources long after its relevance has waned. Meanwhile, newer teams working on emerging priorities are left fighting for scraps. Over time, this approach creates an organization optimized for yesterday’s challenges, not tomorrow’s opportunities.</p><p>The alternative often seems like <strong>demand-weighted allocation</strong>, which shifts resources to wherever the fires are hottest. It feels responsive and logical — if a team is overwhelmed with requests, give them more people. If stakeholders are clamoring for support, prioritize their needs. But firefighting is reactive by nature. It solves the immediate problem while ignoring the structural issues that caused it in the first place. Foundational work — like improving infrastructure, reducing tech debt, or investing in long-term capabilities — rarely generates loud complaints, so it gets overlooked. Over time, demand-driven allocation guarantees that the same fires will keep breaking out, often bigger than before.</p><p>These approaches fail because they try to turn an inherently strategic question into a procedural one. They treat headcount allocation as a way to avoid decisions rather than as an opportunity to make them. Historical allocation dodges the hard conversation about what’s no longer important, and demand-weighted allocation avoids the even harder conversation about what should be. Both approaches are attractive because they feel simple and neutral. But they are neither.</p><p>The allure of simple answers is especially strong in large organizations, where the complexity of competing priorities can feel overwhelming. Simplicity provides an escape: copy what worked before or triage the loudest complaints. But simplicity also hides the trade-offs that make allocation such a high-stakes decision. And in that invisibility lies the real risk — not just that resources will be misallocated, but that no one will notice until it’s too late.</p><h3>A Thought Experiment: The Cost of Imbalance</h3><p>Imagine two companies.. The first allocates 100% of its headcount to today’s priorities. Its core products thrive, but it neglects the foundational work — technical, creative, or cultural — that keeps a company relevant. Over time, its growth slows, and by the time its competitors’ bets start paying off, it’s too late (see IBM, Intel or Blockbuster).</p><p>The second company does the opposite. It invests too heavily in innovation, starving its core products of resources. The result? It becomes a company of ideas without execution, a portfolio of possibilities with no profits to fund them. Stitchfix is an example of this cautionary tale. It’s a company that early in my career was considered the best place for Data Science and Machine Learning, often pushing on the cutting edge boundry way ahead of other tech companies. But when you look at how that innovation translated to execution or delivery of business results, the outlook looks bleak, with all fundementals pointing to a consistent decline.</p><p>Both companies fail for the same reason: they misunderstand the nature of trade-offs. Headcount allocation isn’t about choosing between today and tomorrow — it’s about balancing them. This balance works because it forces clarity on priorities. Businesses that focus solely on execution — without making room for foundational or exploratory work — risk stagnation. Those that focus on innovation, risk running out of money.</p><h3>The Hybrid Model: Strategy Meets Optionality</h3><p>If there’s a better way to approach headcount allocation — and I believe there is — it starts with acknowledging that simplicity isn’t the goal. The goal is clarity. Clarity about what matters now, what might matter later, and what the company is willing to risk. That clarity requires a model that embraces trade-offs rather than avoiding them, balancing the need for focus with the need for flexibility. At its core, that’s what the <strong>hybrid model</strong> is designed to do: allocate most resources to today’s priorities while reserving space for tomorrow’s possibilities.</p><p>The hybrid model works by combining two principles. The first is <strong>business-driven allocation</strong>, which ensures that the majority of resources — 80 to 90% — go to the company’s most pressing needs. These are the bets the company is willing to stake its reputation on, like scaling a product line, entering a new market, or solving a critical operational bottleneck. Business-driven allocation ties headcount to measurable outcomes, forcing teams to justify their requests not in terms of tradition or urgency but in terms of impact. It asks,<strong> “How does this team move the needle for the company today?”</strong></p><p>The second principle is the <strong>reserved innovation pool</strong>, which protects the work that isn’t tied to immediate business metrics but is vital for long-term growth. This might include moonshot projects, exploratory R&amp;D, or foundational investments in infrastructure and tooling. The reserved pool — typically 10 to 20% of headcount — isn’t a free-for-all. It’s managed deliberately, with a clear process for evaluating proposals and a commitment to accountability. The goal isn’t to hedge against uncertainty but to ensure the company remains adaptable, always capable of seizing opportunities or responding to change.</p><p>The power of the hybrid model lies in its balance. Business-driven allocation creates focus, directing the bulk of resources to the initiatives that matter most today. The reserved pool creates optionality, making room for the ideas and experiments that might define the company’s future.</p><h3><strong>Isn’t the 10% to 20% innovation pool too small?</strong></h3><p>At tech companies, where engineers are surrounded by cutting-edge tools and the promise of endless possibilities, the gravitational pull toward innovation can feel irresistible. I once heard a senior engineer say, half-jokingly, “The easiest way to get attention here is to slap ‘AI-powered’ onto an idea and call it innovation.” It wasn’t just a joke — it was a truth about how ambition can sometimes veer into vanity. When the tools, talent, and appetite for invention are this abundant, the risk isn’t that people won’t innovate. It’s that they’ll innovate too much, often on things that don’t matter.</p><p>Consider the scenario: a team is tasked with improving onboarding for a subscription-based product. Instead of focusing on the core issue — simplifying the user flow to reduce friction — they pitch an AI-driven onboarding bot. It’s technically impressive and promises to “revolutionize” the user experience. But months in, the bot isn’t delivering results, and the engineering capacity devoted to it has come at the expense of tackling obvious usability problems that could have moved the needle immediately. This is how innovation, untethered from strategy, starts to erode the company’s ability to execute.</p><p>The challenge here is that innovation feels exciting, while core business work can feel mundane. Engineers want to push boundaries, not debug onboarding forms. Without guardrails, even a 10–20% innovation allocation can balloon into 40–50% of real capacity, as engineers chip away at side projects, experiment with speculative ideas, or stretch innovation efforts well beyond their initial scope. The intention — to motivate and challenge talented people — is good. The execution, however, can lead to what a former manager once called “innovation theater”: flashy projects that generate internal buzz but fail to deliver meaningful business outcomes.</p><p>To fix this, companies need to stop worshipping innovation as an end in itself and start treating it as a means to a larger goal. The best ideas are the ones that are both creative and connected to business strategy. If an innovation project can’t answer basic questions — “What problem does this solve?” “How does it drive impact?” “Why now?” — then it doesn’t belong on the roadmap, no matter how futuristic or impressive it looks.</p><p>More importantly, leaders need to reward the engineers who tackle the unglamorous work. Fixing a leaky funnel or optimizing server reliability might not win awards, but it’s the kind of work that keeps the lights on — and often delivers more value than a dozen speculative prototypes. The future of a company isn’t built on big ideas alone; it’s built on a balance between boldness and practicality. That balance doesn’t happen by accident — it happens when leaders make it clear that innovation isn’t about being “two years ahead of the roadmap.” It’s about making sure the roadmap itself is worth following.</p><h3><strong>Brass Tacks: How to Implement the Hybrid Model</strong></h3><p>Implementing the hybrid model in practice means translating its principles into a structured, actionable process. This involves balancing quantitative rigor, like evaluating ROI or business impact, with qualitative judgment about strategic priorities, team capabilities, and the root causes of inefficiencies. Beyond just deciding where to add resources, this process should also question why existing headcount isn’t achieving more and address systemic issues like tech debt or team inefficiencies. Here’s how to approach it:</p><h4><strong>Step 1: Ask: Why Can’t This Team Achieve More with Its Current Headcount?</strong></h4><p>Before allocating additional resources, take a step back and assess whether the team is truly constrained by headcount — or by deeper structural issues that limit its capacity. Are they tackling the right priorities, or are they bogged down in low-impact tasks? Are they spreading themselves too thin across competing projects? A team that can’t clearly articulate its roadmap or justify its priorities may not need more people — it may need better focus. Dig deeper: Is the team’s productivity hampered by technical debt or architectural complexity? For instance, brittle systems, patchwork solutions, or poorly thought-out dependencies can create overhead that consumes disproportionate amounts of time and effort. Instead of rushing to add headcount, consider whether resolving these blockers — through better tooling, process improvements, or foundational investments — could unlock the team’s existing potential. Sometimes, the most effective solution isn’t adding more people to the fire; it’s putting out the fire itself.</p><p>During this process, I would also evaluate the team track record. If a team has a consistent track record of strong execution and delivering results, I would consider assignning more resources to them and expanding their scope instead of allocating headcount to less successul teams.</p><h4><strong>Step 2: Start with Clear Business Priorities</strong></h4><p>Headcount allocation must be anchored to the company’s strategic goals for the year, ensuring resources flow to the initiatives that deliver the highest impact. For example, if a SaaS company’s focus is on reducing churn, increasing customer acquisition, and improving operational efficiency, these priorities should directly shape how the <strong>90% business-driven allocation</strong> is distributed. But aligning headcount with business goals doesn’t mean waiting for teams to request it — it means proactively identifying where resources are needed, even if the team hasn’t flagged it.</p><p>Let’s say the company’s goal is to improve infrastructure cost efficiency by 50%. Headcount allocation should target teams working on projects directly tied to this outcome, whether they’ve explicitly requested headcount or not. For instance, if the infrastructure team has a roadmap to optimize cloud costs by automating workload scheduling or reducing over-provisioning, additional engineers should be allocated to accelerate those efforts. However, this doesn’t mean simply rewarding teams that are vocal about their needs. Instead, leadership should evaluate every team’s roadmap against the stated goal and allocate headcount to the initiatives with the clearest path to impact — whether or not the team has flagged resource constraints. This is where leadership judgment plays a critical role: separating teams’ perceived needs from what the business truly requires.</p><p>To describe this more clearly. If my goal is to save $100M in infrastructure costs, I would lietrally go across team roadmaps and divide the benefit/eng effort to understand which teams can bring me closer to that goal. Similarly, if my goal is to increase GOV by $1B/year, I would do the same exercise that divides the benefit/eng effort to get the teams that would bring me closer to the goal.</p><h4>Step 3: Evaluate the Team’s Roadmap</h4><p>Before allocating additional headcount, scrutinize the team’s roadmap. A strong roadmap doesn’t just outline high-priority projects — it provides a realistic, actionable plan for delivering measurable impact. Teams with clear milestones, well-defined deliverables, and alignment to business goals are better candidates for additional headcount than those with overly ambitious or reactive plans. However, roadmaps can sometimes include padded numbers or inflated promises that look impressive on paper but don’t translate into real-world results. Leadership must learn to identify and address these issues to avoid misallocating resources.</p><p>To deal with padded roadmaps, dig deeper into the assumptions behind the numbers. For example, if a team claims that a new feature will increase revenue by 20%, ask how they arrived at that figure. What data supports this projection? Has a similar initiative delivered comparable results in the past? If the estimates rely on best-case scenarios without accounting for risks or dependencies, leadership should push back and request more grounded expectations. Similarly, look for vague or unverifiable claims — such as “enhanced user experience” — that lack clear metrics or defined outcomes. If the roadmap fails to connect these claims to tangible impact, it may not justify additional resources.</p><p>One way to validate roadmaps is by cross-referencing team performance on past initiatives. Did the team meet previous targets, or were their projections overly optimistic? A track record of consistent execution lends credibility to their current plan, while a history of missed goals should prompt further scrutiny.</p><p>Finally, encourage teams to adopt a culture of honest planning by rewarding transparency. Teams that present realistic, thoughtful roadmaps — even with modest numbers — should be prioritized over those that inflate projections to compete for resources. By creating a system where honest roadmapping is valued, leadership can ensure that headcount allocation supports meaningful, achievable goals rather than paper successes.</p><h4>Step 4: Define and Protect the Innovation Pool</h4><p>The innovation pool is a crucial part of the hybrid model, ensuring that teams have the capacity to explore new ideas, address long-term challenges, and seize future opportunities. However, without clear boundaries and processes, innovation efforts can quickly become scattered, misaligned, or deprioritized. Or they can quickly baloon to consume 50%+ of the engineering capacity, as I’ve often observed. Defining and protecting the innovation pool is about creating the structure and discipline needed to balance creativity with accountability.</p><p>Start by clearly allocating 10–20% of headcount to innovation across the organization. Depending on the company’s size and structure, this can be centralized (a dedicated team working on high-impact speculative projects) or decentralized (individual teams reserving a portion of their capacity for innovation). If the pool is decentralized, ensure that innovation time is formally built into team roadmaps rather than being treated as a leftover resource for “when there’s time.”</p><p>Leadership should also require teams to justify their innovation projects with concrete proposals. What problem is this idea solving? How does it align with broader business goals? What would success look like? By setting these expectations upfront, companies can ensure innovation efforts are strategic rather than speculative.</p><h3>Conclusion: Allocating Ambition</h3><p>Headcount allocation is one of the most critical decisions a company makes, yet it often feels like an afterthought — dominated by politics, spreadsheets, and reactive firefighting. But at its core, headcount allocation isn’t just about numbers; it’s about ambition. It’s about deciding what a company prioritizes, what it’s willing to risk, and, ultimately, what it wants to become.</p><p>The hybrid model offers a way forward. By allocating the majority of resources to today’s priorities while reserving space for tomorrow’s possibilities, it strikes a balance between execution and exploration. It forces clarity, turning headcount allocation into an act of strategy rather than a procedural chore. It challenges us to invest in what matters most while creating room for the ideas that might shape the future.</p><p>But the model is only as good as the discipline behind it. Implementing it requires leadership to dig deeper: questioning why teams aren’t achieving more, scrutinizing padded roadmaps, and protecting the innovation pool from the gravitational pull of immediate demands. It demands honesty, accountability, and the willingness to make hard trade-offs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af3400ef3fc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why has Snowflake or Databricks not acquired DBT?]]></title>
            <link>https://stassajin.medium.com/why-has-snowflake-or-databricks-not-acquired-dbt-aeb01168c5e4?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeb01168c5e4</guid>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 01:30:29 GMT</pubDate>
            <atom:updated>2023-09-14T01:30:29.093Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Disclaimer</em></strong><em>: This article is speculative and based on personal opinions. As of the time of writing, there is no indication or news confirming that Snowflake or Databricks plans to acquire DBT Labs.</em></p><p>In this post, I’m trying to understand why given DBT’s and Databricks/Snowflake&#39;s shared customer base and the jigsaw-like fit of their capabilities an acquisition has not happened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*J499NSovgXb_Tyt9Qfxozw.png" /><figcaption>DALL*E <a href="https://labs.openai.com/e/aEF4M1RPRFjyCmWQIlgxo8lP/AVehkbXDb8YSfCxtdgXC8Lbl">interpretation</a> of “Marie Curie writing a DBT model on a computer”</figcaption></figure><h3>DBT’s Pricing Changes: A New Paradigm</h3><p>DBT Labs recently moved to a consumption-based pricing model, a departure from its seat-based pricing. This shift makes sense for several reasons:</p><ol><li><strong>Sustainability</strong>: We should dispense with the idea that open-source software is some altruistic endeavor that exists outside the realm of economics. Companies need to keep the lights on. And while DBT Labs has garnered a cult-like following for its open-source contributions, goodwill and impractical pricing models don’t make payroll.</li><li><strong>Value Realization</strong>: As DBT becomes the go-to platform for a growing list of data operations — from transforming data to validating it — the disconnect between the value provided and the revenue generated has widened. This change is an attempt to correct that balance, and it’s hard to argue against a company seeking fair compensation for the utility it delivers.</li></ol><h3>Snowflake and Databricks: The Elephant (or Elephants) in the Room</h3><p>It’s tough to discuss DBT without mentioning Snowflake or Databricks. The latter have become more than just tools; they’re platforms that underpin entire analytics ecosystems. So how does DBT’s pricing change impact these giants?</p><p>It’s a simple but potent change. Raise the cost per query in DBT, and you introduce a pause, a moment of hesitation before users hit ‘Run.’ While this newfound frugality may seem inconsequential, it’s not. Each query not run is a compute cycle not spent, a piece of storage not used. It’s an invisible shrinking of the boundary conditions within which Snowflake and Databricks operate.</p><p>If you’re wondering whether this impacts the financial health of Snowflake and Databricks, the answer is an unambiguous yes. These platforms derive value not just from hosting your data but from the myriad ways you interact with it. Fewer queries and reduced storage are not simply less activity; they translate into less revenue. And revenue, as they say, is the lifeblood of any business — be it a start-up or a juggernaut.</p><h3>The Upside: When DBT Meets Snowflake and Databricks, Synergy isn’t Just a Buzzword</h3><p>But let’s not paint a one-sided picture here. If DBT’s pricing changes introduce constraints, the tool’s seamless integration with Snowflake and Databricks also uncorks a lot of value. So, what does this synergy — yes, an overused term but genuinely applicable here — really offer?</p><h4>The Transformation Engine: It’s Mostly DBT</h4><p>In many organizations, DBT is the go-to workhorse for data transformation. Its unique capabilities are not just an added value; they are often a compelling reason for the adoption of underlying platforms like Snowflake and Databricks.</p><h4>Data Validation: DBT Adds Depth</h4><p>Then comes the notion of data quality, an area where DBT excels with its SQL-based checks. With DBT in the mix, you’re not just storing data in Snowflake or processing it with Databricks — you’re also validating its quality. This added layer of interaction increases the platforms’ stickiness and, frankly, makes DBT more indispensable to daily operations.</p><h4>Metrics and More: Third-Party Leverage</h4><p>Don’t underestimate the power of third-party tools like Hex or Mode, which leverage DBT’s semantic layer to offer enriched analytics. While they may seem like peripheral players, they often encourage a higher volume of queries. This has a cascade effect: the more queries run, the more Snowflake and Databricks come into play, bolstering their usage and, ultimately, their revenue streams.</p><p>When you factor in the three reasons above, it feels like a complete integration would be a step that would benefit either of the big players.</p><h3>Why No Acquisition?</h3><p>So, why haven’t Snowflake or Databricks made the move to acquire DBT Labs? So far I can only think of these two reasons:</p><ol><li><strong>Valuation</strong>: DBT Labs has grown in prominence and value, making it a potentially expensive acquisition target. In 2022 it was valued at <a href="https://www.forbes.com/sites/kenrickcai/2022/02/24/dbt-labs-series-d-4-billion-less-than-planned/?sh=d6b687367c3c">over $6B</a>. Although the private valuations compressed since then by a lot, it would still be an expensive acquisition.</li><li><strong>Independent Path</strong>: DBT Labs isn’t just sitting by the phone waiting for an offer. They’ve built a moat — a ubiquity in the data world — that suggests they might have grander plans of their own. The idea of charting an independent course and embracing the challenge of monetization could be more enticing than becoming a subsidiary.</li></ol><p>Despite these hurdles, the complementary nature of these technologies makes the acquisition question incredibly intriguing. The recent pricing changes by DBT Labs make this all the more relevant, potentially affecting both its and its would-be acquirers’ revenue streams. Whether or not an acquisition is on the horizon, the evolution of DBT as a company will undoubtedly be interesting to watch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeb01168c5e4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The downsides of experimentation]]></title>
            <link>https://blog.devgenius.io/the-downsides-of-experimentation-e3be2545e124?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3be2545e124</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 00:33:11 GMT</pubDate>
            <atom:updated>2023-09-12T17:54:01.401Z</atom:updated>
            <content:encoded><![CDATA[<p>Experimentation is the tech world’s equivalent of a gym membership. Everyone wants to sign up, but mastering how to run experiments demands a lot of effort. On the surface, experimenting seems like an obvious and simple formula: make small changes, measure, iterate. However, this post aims to delve into the less discussed, yet critical, aspects of experimentation and outline elements that, if neglected, can make you look as though you’re merely rearranging deck chairs on the Titanic.</p><p><strong>I want to be perfectly clear: </strong>this post isn’t an argument against experimentation — quite the opposite. I’m confident that on average companies that rely on experimentation are more successful than those that use other forms of decision making. Instead, consider this post as an invitation to enrich your understanding and practice of experimentation, incorporating it into a broader strategy for success.</p><h3>Paper-cuts</h3><p>I would like you to consider the following thought experiment. Imagine you have three groups:</p><ul><li><strong>Group 1</strong>: Experiences a refined version of the product, improved based on previous successful experiments.</li><li><strong>Group 2:</strong> Subjected to constant changes due to ongoing experiments, from new features to design alterations. Group 2’s user experience is akin to living in an architectural fever dream — nothing stays the same. Instead of signing these users up for a reliable service, we’re taking them on a roller coaster.</li><li><strong>Group 3</strong>: Interacts with a stable version of the product, updated only for bug fixes.</li></ul><p><strong>Which group do you think is likely to have the most optimal experience?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TUDVVtO5tztsNbuJN9W4xA.png" /></figure><p>While Group 1 enjoys a premium experience and Group 3 gets a consistent, albeit static, user experience, Group 2 may actually find themselves in the least enviable position. They might be constantly thrashed by paper cuts manifesting as ever-changing features, inconsistent interfaces, terrible ideas, and general product unpredictability. Each individual change might be minor, but the sum total can lead to a frustrating and disjointed experience. Moreover, a high frequency of changes not only affects user experience but also increases the risk of introducing technical issues. This risk could manifest in the form of significant regressions or even major system incidents, potentially causing the Service Level Agreements (SLAs) for Group 2 to be the worst among the three groups.</p><p>Note that the lesson here is not to completely halt product evolution by “freezing” it; rather, the challenge is to continually improve the product while mitigating the downsides that constant changes can bring, something that is very, very difficult to do.</p><h3>Snacking</h3><p>One of the paradoxical downsides of a culture of experimentation is that it can sometimes encourage an overly cautious approach to innovation, known colloquially as “snacking.” In this process, individuals or organizations make small, incremental changes to an existing system or product, optimizing for short-term gains or local maxima. While these adjustments might yield positive results in the immediate context, they often discourage people from taking bolder, more substantial risks that could lead to significantly greater rewards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_yoDLXGEua9imY63.jpg" /><figcaption><a href="https://www.intercom.com/blog/first-rule-prioritization-no-snacking/">Source</a></figcaption></figure><p>What’s particularly challenging is that this culture of experimentation often goes hand in hand with a culture of “scorecard chasing”, where everything is about generating impact. Note that on average this is a good thing, but when taken to an extreme it manifests itself with a lack of risk-taking at the IC level: “Why opt for a riskier project that might not show results by my next review?”</p><p>So, how can we reap the benefits of experimentation while also leaving room for more drastic forms of innovation? I don’t have a good answer, but part of me is very fond of the idea of having autonomous units described in the archipelago model below.</p><p><strong><em>Archipelago Model</em></strong></p><p>Let’s lean into this idea of autonomous units. Think of it like a corporate archipelago: you’ve got a bunch of different islands (or teams), each with its own microclimate and species of flora and fauna. In one corner, you’ve got your well-established, “big and consolidated product” island. Here, the name of the game is ‘incremental maximalism’ — a fancy way of saying we’re optimizing the hell out of what we already have. Now, hop on a metaphorical boat and sail over to the “new offshoots” island. This is your R&amp;D lab, your skunkworks, your “what if we tried this crazy thing” territory. Here, you don’t get a pat on the back for a 0.5% gain; you get it for demonstrating that a radical new idea has legs. The risk is higher, sure, but so is the potential payoff. Failure isn’t just accepted; it’s expected.</p><p>So why does this archipelago model make sense? Because it allows a single organization to wear different hats simultaneously. One hat for “We need to improve what we’re already doing” and another for “Let’s shoot for the moon and try something that could be game-changing.” And because these units are autonomous, it becomes easier to specify separate incentive structures and cultures. In this model, you can’t have the R&amp;D department reporting to the CFO who wants to optimize for 2% more revenue in the next quarterly call.</p><p>This is obviously a simplification, given that collaboration and cross-pollination between the islands is essential. You can’t just have a team doing deep tech for decades without having a strategy for how that would consolidate and grow your moat.</p><h3>Replacing dashboards with experiments</h3><p>When running an experiment, it’s common to define narrowly focused metrics that measure the specific impact of an initiative. These metrics are useful for understanding how well the experiment is performing in relation to its predefined objectives. However, a problem arises when teams perform their impact reporting purely in the context of experiments.</p><p>Let’s go through a few scenarios to understand why it is important to look at real metrics rather than only on reported cumulative experiment wins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EcRpi2mkya0Xyvf2x3kEQ.png" /><figcaption>Different scenarios for how your real metric might look in contrast to the reported experiment “wins.”</figcaption></figure><ul><li><strong>Scenario 1</strong>: The ideal scenario is when the growth in the real metric matches with the reported growth from experiment changes. If you report an absolute change of 10% in conversion due to your experiments, most people in your company will have the expectation that there will also be a positive change in the company metric matching your impact.</li><li><strong>Scenario 2</strong>: In this situation, you report experimental gains while overall business performance metrics remain unchanged. This disparity can raise legitimate concerns about the accuracy of your experiment’s claimed success.</li><li><strong>Scenario 3</strong>: Here, you report experimental successes even as the company’s overall metric is on the decline. Such a discrepancy often suggests that there are fundamental issues requiring immediate attention; relying solely on experimental metrics in this case can be misleading.</li></ul><p>The discrepancy I’m explaining above is not as rare as some might assume. For example, in many companies, the marketing and sales budgets are predefined ahead of time. If you find a way to save 20% on those costs because of more efficient spending, you might find that folks controlling those budgets will immediately spend the surplus on other areas that are similarly cost-inefficient, netting you a zero impact.</p><p>In summary, while narrowly focused experiments offer critical insights, it’s crucial to corroborate these findings by examining them in the context of more comprehensive, company-wide performance metrics. This holistic approach provides a more accurate representation of how well both the experiment and the organization are truly performing.</p><h3>Dead Code</h3><p>Consider this scenario: your organization runs 5,000 experiments a year, and even if just 20% of them don’t make it to production, that’s 1000 dead ends, each potentially leaving behind traces of code that serve no purpose. This can include unused functions, irrelevant variables, and deprecated feature flags. Now, multiply this by several years of operation, and you have a sizable labyrinth of dead code paths and branches cluttering your codebase. The time and resources needed to clean up this ‘digital detritus’ can be considerable and distract from value-adding activities.</p><p>The problem is that experimentation significantly amplifies the rate at which this dead code accumulates. Each experiment that doesn’t pan out adds to the growing pile, making the codebase more difficult to navigate, understand, and maintain. Note that dead code is not dissimilar to software cruft that Martin Fowler describes <a href="https://martinfowler.com/articles/is-quality-worth-cost.html">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*jZIXC5eW39ho2Tnz.png" /><figcaption><a href="https://martinfowler.com/articles/is-quality-worth-cost.html">Source</a></figcaption></figure><h3>Conclusion</h3><p>In wrapping this up: experimentation isn’t the silver bullet some make it out to be. Don’t get me wrong — I’m not waving a flag for us to retreat from the front lines of A/B tests and iterative changes. All I’m pointing out is that experimentation has its downsides when it’s taken to an extreme. Some are subtle, like the constant paper cuts of never-ending tweaks that might erode the user experience. Some are more profound, like the risk of “snacking” your way into a local maximum while ignoring game-changing opportunities.</p><p>I’m all for leaning into the scientific method to make products better. But we owe it to ourselves — and our users — to remember that experimentation isn’t the be-all and end-all. It’s a tool in a broader toolkit, and like any tool, its utility is defined by the skill and wisdom with which it’s used.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3be2545e124" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/the-downsides-of-experimentation-e3be2545e124">The downsides of experimentation</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High-leverage Activities for Engineering Managers]]></title>
            <link>https://medium.com/better-programming/high-leverage-activities-for-engineering-managers-b919bf61f26c?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/b919bf61f26c</guid>
            <category><![CDATA[management]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Mon, 14 Aug 2023 15:42:42 GMT</pubDate>
            <atom:updated>2023-08-21T14:43:30.064Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zfIO91bkv9Wqxyqp" /><figcaption>Photo by <a href="https://unsplash.com/@anniespratt?utm_source=medium&amp;utm_medium=referral">Annie Spratt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every manager has the same number of hours in a day, yet some consistently achieve better results, foster innovation, and cultivate teams that are both happy and highly productive. What sets these managers apart? The secret often lies in their commitment to high-leverage activities.</p><p>As you navigate the labyrinth of management challenges, always remember that every individual you lead is an investment in the future. By focusing on high-leverage activities, you’re not just leading a team but building an enduring legacy of excellence, collaboration, and innovation. It’s crucial to remember that the true measure of success is not just in the achievements of the leader but in the collective triumphs and growth of their team.</p><h3>Focus on top performers</h3><p>Your life and career as a manager will be much more successful if you hire and retain great individuals on your team. It’s not uncommon to see managers with mediocre technical and strategic thinking succeed rapidly through their careers because they were able to hire, inspire, and also keep individuals that are simply amazing at execution. Top performers aren’t just assets; they are catalysts. They not only bring exceptional skills and passion but also elevate the entire team’s performance by setting high standards and mentoring others. Moreover, recognizing and appreciating their efforts is paramount. It not only fosters loyalty but also motivates them to push boundaries and innovate.</p><p>One common mistake I see is conflating dependency on an individual with top performance: “We need to keep Don because he is the only one who knows how this stuff works.” If you’re in this position, I’m sorry for you. I would encourage you to rapidly find a solution and to also design the right culture and processes in place that prevent you from being in a position of vulnerability. Focus on individuals that can drive impact while they also try to make themselves replaceable, not on those who keep you hostage.</p><h3>Meetings</h3><p>Paul Graham wrote insightfully about the aversion programmers have towards meetings. It’s often observed that meetings come at a higher cost for engineers, and the value they extract is often questionable. A root cause I’ve pinpointed is the inefficiency of how many meetings are conducted. Reflect on this: in a sizable number of meetings that bring together managers and individual contributors (ICs), the conversation is dominated by managers, often capturing more than 50%, and in some cases, even 80% of the total speaking time. This stark imbalance inadvertently muffles the voices of ICs, rendering their perspectives underrepresented or entirely unheard.</p><p>Now, if you’re a manager, let’s grapple with a thought-provoking question: “How can you truly understand the challenges your team faces or steer them towards a conclusive decision if you are perpetually in the spotlight, doing the bulk of the talking?”</p><p>This is why I think listening is a high-leverage activity and I encourage you to embrace active listening, set clear agendas, allocate time to speakers and encourage participation so that different perspectives are heard.</p><h3>Align the team through metrics</h3><p>I read an apocryphal story in <a href="https://www.amazon.com/Homo-Deus-Brief-History-Tomorrow/dp/0062464345/ref=pd_lpo_sccl_1/143-3601736-6784032?pd_rd_w=NtRjA&amp;content-id=amzn1.sym.116f529c-aa4d-4763-b2b6-4d614ec7dc00&amp;pf_rd_p=116f529c-aa4d-4763-b2b6-4d614ec7dc00&amp;pf_rd_r=GCQZTHV8MXTFTFSF4GXR&amp;pd_rd_wg=eUhu3&amp;pd_rd_r=0049fb8f-1ef0-408c-8a8a-f97e2102d99f&amp;pd_rd_i=0062464345&amp;psc=1">Homo Deus</a> about a group of Soviet economists visiting the UK, to exchange and compare economic theories. The hosts took the Soviets to a tour of a city, of the London Stock Exchange and the London School of Economics where they talked in length about economic policies. After a few hours, a Soviet expert burst out: “Wait a moment, please. Let’s table all the complicated economic theories. We’ve been through London back and forth for a full day and there is one thing I cannot understand. Back in Moscow, our finest minds are working on the bread supply system, and yet there are queues at every bakery. Here in London live millions of people yet I have not seen a single bread line. Please take me to the person in charge of supplying bread to London.”</p><p>There is obviously no central authority that decides how much bread should be produced in a given day and that it should be priced everywhere at £2/loaf. Instead, we rely on the “invisible hand” and economic incentives to drive and align millions of agents. In a similar vein, there is no single project or solution that will guarantee success in your company or career. Instead, you want to align people by pointing them to the north star and operational metrics and asking them to shape their own destiny. There are several second-order advantages to doing this:</p><ul><li><strong>It helps you walk backward</strong>: Individuals and teams will have an easier time walking backward from the metric to the problem and then to the solution.</li><li><strong>Velocity</strong>: You&#39;re more likely to encourage bottom-up thinking and ideation where team members can move much faster since they have a deeper understanding of the problem and what an optimal solution would look like.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O36WnCWwT7npgc0qk_418Q.png" /></figure><p>As a final piece of advice on the north star: don’t pick something performative that only your team cares about (tickets closed, tables created, APIs developed, etc). Pick something real, something that can be measured in real economic impact, or something that everyone in the company cares about (e.g., headcount toil reduced, costs, revenue generated).</p><h3>Hire for diversity</h3><p>I believe many companies are giving up a large amount of value by being too standardized in how they hire engineering talent. If every team member is uniformly ‘good’ at everything but doesn’t excel in any particular area, the team might lack the diversity of thought and skill needed to innovate. On the other hand, a team composed of members each with their distinct strengths can be more dynamic, versatile, and creative.</p><p>Take, for example, a company that primarily hires based on Leetcode performance. Over time, it might find that its team increasingly consists of individuals who prioritize such skills. This can create a feedback loop where technical interviews revolve predominantly around Leetcode challenges. Similarly, if a data science team is predominantly filled with statisticians, it’s highly probable that subsequent hires will mirror this profile, offering little in terms of diversifying the team’s knowledge base.</p><p>Individuals with pronounced strengths in specific areas can often provide more value than those who are just ‘good’ across the board. For instance, if you’re hiring for an engineering position, a candidate who’s exceptionally skilled at API design but weaker in algorithms might drive significantly more impact, compensating for the minor shortfall in another area. The important trick though is that you need to hire for the strengths your team lacks, which does require more planning and effort in defining your hiring goals.</p><p>Think of it this way: rather than hiring the 50th engineer who knows how to Leetcode, why not hire an expert who knows databases really well and hyperfocus on assessing those skills? That way you can finally resolve some of your current database scaling problems that lead to innumerable P0s.</p><h3>Have good onboarding and training</h3><p>Andy Grove, the former CEO of Intel and author of “High Output Management” gave a very interesting mathematical justification for having good training and onboarding. As a manager, if you spend an extra hour on the job, you might get an extra hour of work done. However, if you spend that hour training someone else, that person might save an hour doing something more efficiently every day for months to come. If your training is done in a collective context where your advice and lessons can reach more people, you’re effectively getting very good leverage with hundreds of hours of productivity gained from less than a dozen hours of investment.</p><p>Unfortunately, most companies don’t do onboarding well. As a general rule, what I’ve seen is that folks put together an onboarding doc with general resources, and schedule a 1:1 once a week to answer questions between the manager (or onboarding buddy) and new hire. That’s a good start, but things could be improved. I found that the following onboarding tricks provide high leverage.</p><ul><li><strong>Schedule daily 1:1</strong> with the new hire to help them answer ANY questions they have. I usually keep the daily cadence for two weeks for senior people and up to one month for more junior folks. I can often tell if a candidate will become successful within two weeks not based on the questions they ask but on the quantity of questions. If they tell me on Day 5, “I don’t have any questions,” that often tells me that they are not learning enough, which is a very unusual thing to happen when you first join a company.</li><li><strong>Well-defined starter project</strong>: set them up with a starter project that is very well-defined in scope and complexity and that can be delivered within one month. This project should also give them the opportunity to spin up new initiatives or offshoots. It shouldn’t be a “fix a bug” project or a project that is effectively a dead-end once it is completed. It should be something meaningful, impactful, and doable.</li><li><strong>Trim down the noise</strong>: Organizations generally have too much documentation that is outdated or irrelevant. I would also be the filter that can immediately point them out to relevant and high-quality documentation or can connect them to other people in the org.</li><li><strong>Pair program: </strong>I would often pair program with them using live coding functionality. This allows me to guide them through the codebases in more detail, often with a debugger, and also gives them a glimpse into what my local setup looks like so that they can mirror it.</li><li><strong>Levels expectation</strong>: Lastly, I would literally open the levels expectation doc and walk them through it. It can be very discouraging if you join a company and nobody tells you how performance is measured or what good performance means. Being upfront about expectations is more likely to lead to alignment and “no surprises” during the review period.</li></ul><h3>Conclusion</h3><p>The role of an engineering manager extends beyond just supervising tasks and processes. It’s a blend of strategic thinking, fostering inclusivity, recognizing potential, and constantly innovating on best practices. Effective engineering managers recognize that their impact is magnified by enabling their teams to thrive. Prioritizing high-leverage activities — from harnessing the power of top performers, streamlining meeting dynamics, using metrics as a compass, and championing diversity in hiring, to investing in robust onboarding and training — can make a significant difference.</p><p>Ultimately, the goal of every engineering manager should be to curate an environment where each team member feels acknowledged, understood, and empowered to take the reins of their own growth. By doing so, not only is there a surge in innovation and productivity, but it also fosters loyalty, talent retention, and consistent team performance. It’s crucial to remember that the true measure of success is not just in the achievements of the leader but in the collective triumphs and growth of their team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b919bf61f26c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/high-leverage-activities-for-engineering-managers-b919bf61f26c">High-leverage Activities for Engineering Managers</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B Testing: How Misaligned Incentives Can Sabotage Your Experimentation Culture]]></title>
            <link>https://stassajin.medium.com/a-b-testing-how-misaligned-incentives-can-sabotage-your-experimentation-culture-b5a37cc252d7?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5a37cc252d7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Fri, 24 Feb 2023 22:59:40 GMT</pubDate>
            <atom:updated>2023-02-27T02:07:10.917Z</atom:updated>
            <content:encoded><![CDATA[<p>A/B testing, also known as split testing, is a popular method used by companies to improve the performance of their products. It involves comparing two versions of a design, feature, or content to determine which one performs better. Despite its popularity, A/B testing programs often fail to deliver the intended impact.</p><h3>Misaligned incentives</h3><p>Most experiments you run test a change to which you have a personal attachment. You are ultimately trying to increase some number, receive a big congratulatory pat on the back, and feel validation for a job well done. Will you truly be a passive observer, a good scientist, and simply accept restarting the work if the numbers tell an uncomfortable story? Will you truly be impartial, not influenced by incentives and recognition? Are you able to set aside that uncomfortable feeling when you inform your stakeholders about the time and resources that were wasted? I don’t think so.</p><p>In academia, there is a strong incentive to “publish or perish”, and there are an untold number of stories related to questionable research practices and even fraud. One thing that folks don’t discuss enough, is that in the industry, that cultural phenomenon is probably worse since incentives are a lot more short-term and real, pressure is higher, and people might be less scrupulous and more prone to misuse statistics. Due to misaligned incentives, the result of any experiment can be adjusted to make anything look significant. For example, <a href="https://journals.sagepub.com/doi/10.1177/0956797611417632">Simmons et. al (2011)</a> did a simulation where they show that simply by having a choice among four possible analysis decisions, anything could be made to look stat sig:</p><ul><li>Testing for two correlated metrics, but reporting only on the one that looks better.</li><li>Peeking sequentially at data.</li><li>Adding a covariate.</li><li>Trying three variants and reporting only on the successful one.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTl1tWBa4NSzj3IxLogo6w.png" /><figcaption>False Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.</figcaption></figure><p>I can tell you that these four choices are just a few of the many options an analyst has when examining results. I know of a dozen other ways to make results statistically significant: transforming the metric, handling outliers, trying different combinations of CUPED covariates, segmenting data, restarting experiments, trying different statistical methodologies (vanilla OLS vs complex LMM), introducing interference, etc.</p><p>The unfortunate outcome of this flexibility is that you end up with a lot of experiment “wins” that don’t really do anything. Your north star metric will still remain flat year-over-year, despite all the reported cumulative improvements that suggest that you’re increasing revenue by 300%. Moreover, because A/B testing has the appeal of rigor that is shrouded in science and statistics, the golden standard, exceptionally few internal stakeholders will be willing to call out the emperor without clothes. This means that you might have to go through a few years before you call it quits and recognize that something needs to be done to fix the pathology caused by misaligned incentives.</p><p><strong>Solution</strong></p><p>One solution to this problem might be to have a governing body that ultimately has a say in an experiment’s outcome. In other words, “you do the work, we do the analysis and decide the outcome.” Although this might sound like it would work, it&#39;s not a scalable solution and creates far too many organizational bottlenecks to doing work. You might be adding a 30–50% delay to make decisions because you’re waiting for someone to say “ship it.” Moreover, you need and want people to be somewhat invested in outcomes since those are the same people who end up coming up with new ideas and follow-ups to existing experiments. I hardly see a dispassionate centralized authority being creative or even nuanced enough in making ship decisions.</p><p>A solution that I personally prefer, largely due to the peace of mind it provides, is to preregister the experiment design. In writing, I provide all the choices I make about how my data will be analyzed, which metrics I will look at, how outliers will be handled, and even the code that will be used to run the analysis. I personally like this solution because I hate the feeling of thinking of myself as a fraud simply because I engage in posthoc analyses and I don’t remember which analysis I actually planned to use. By writing things down before the test, I can get more precise feedback from stakeholders, improve the quality of my design, and report on effects that are more likely to be real. But even then, I still think most will succumb to peer pressure and revert towards an experimentation culture lacking trust. Your honesty and integrity can easily crumble if those around you get more recognition when they engage in dubious practices.</p><h3>Feeling blocked and bottlenecked</h3><p>One other observation I see across some companies that use A/B testing is that they are running their tests for far too long, leading to a scenario where experimentation capacity acts as a blocker on how much work you can do. Analytics-Toolkit <a href="https://blog.analytics-toolkit.com/2022/what-can-be-learned-from-1001-a-b-tests/">published a meta-analysis</a> on 1000 tests, and the median test duration was 30 days (which is very similar to what many folks report across other companies). This is far too long to get to a decision and creates large swaths of idle time within teams. You have the capacity and desire to iterate on five ideas a month, but because you’re waiting on an A/B test, you have to pick and choose one idea to implement. Moreover, because you put all your eggs in one basket, you’re going to be incentivized and feel more pressure to make sure that the analysis results look “right” and p-hack your way to a successful outcome.</p><p>Personally, I think the majority of A/B tests should take no more than two weeks to run, with the exception of holdouts, tests that are one-way doors, or tests that can create UX thrash. The opportunity cost of not trying new things is just much higher than the cost of occasionally shipping something that doesn’t work as expected.</p><p><strong>Solution</strong></p><p>I think the industry has explored and sort of trodden on the same path of suggesting to use some form of residualized regression (see CUPED, CUPAC, MLRATE, etc). But besides using covariates to reduce your variance, there are a lot more approaches available.</p><ul><li>Frequentist statistics is inherently pessimistic since in many tests you make a tradeoff between false positives and false negatives of 1:4 (i.e, alpha of 5% and power 80%). That’s not an appropriate choice for many experiments that would likely benefit more from balanced error rates. See <a href="https://lakens.github.io/statistical_inferences/errorcontrol.html#justifyerrorrate">Justify Error Rates by Lakens.</a></li><li>Moreover, most hypotheses we ask are directional, yet we still shoot ourselves in the foot by assuming two-sided tests.</li><li>We also often track really bad and insensitive metrics in experiments. MAU as a primary metric for a two-week experiment is unlikely to move much. Simply by being smarter about your metric choice, you can get high sensitivity.</li><li>We also <a href="https://booking.ai/overtracking-and-trigger-analysis-how-to-reduce-sample-sizes-and-increase-the-sensitivity-of-71755bad0e5f">over-track</a> events and don’t really focus on the treated population.</li><li>Or we use an inappropriate design that doesn’t control for observed covariates. <a href="https://www.youtube.com/watch?v=SBwPLwVOb7s&amp;ab_channel=GaryKing">See the fully blocked randomized design by King et. al.</a></li><li>Or use <a href="https://arxiv.org/pdf/2102.13550.pdf">conditional probability or predictive power</a> metrics to identify “lost causes.” Lost causes are experiments where, halfway through your sample size, the results look flat and you should just stop your test and move to the next idea.</li></ul><p>All of the choices above, when implemented reliably, can lead to 10x sensitivity in your experiments, which can materialize in more efficient use of resources and time. Now you won’t feel so bad if one of your ideas fails, because you have four more chances to get to a successful one.</p><p>Some critics might say that this increases the false positive rate, but my response to that is to not think about A/B testing in the context of single experiments. Instead be a <a href="https://en.wikipedia.org/wiki/Markowitz_model">Markowitz</a>, or a gambler who enters a casino and knows that if they have a tiny edge, their payoff across many trials will converge to <a href="https://plato.stanford.edu/entries/paradox-stpetersburg/">infinity</a>.</p><h3>Ignoring new users</h3><p>Meta is well known for their A/B testing program. Although I thoroughly enjoy reading some of their pioneering work in this area, I can’t help but feel that they have reached a point of local optimization where new users don’t get as much coverage as existing users. When I tried to use their products after several years of absence, I just couldn’t onboard to anything because it was all filled with ads or noise. I find that for any consumer-facing product out there, the complexity of the interface just constantly becomes more and more bloated, effectively killing platform growth. Let me give you another example, not to pick only on Meta. Google has increased its ads count on the platform, presumably thinking that the tradeoff between ad revenue and product adoption looks good. But that might be true only for existing users. A user that comes from DuckDuckGo, where they might see half the ads, might find it very difficult to onboard to this experience. Today, for example, I never trust Google with searches that relate to reviews or purchasing decisions because even non-ads look and feel like ads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wGJZyrZBic_YRHggv9Ukvg.png" /><figcaption>My experience on Google</figcaption></figure><p>A/B testing focus is partly to blame for this local optimization. There is probably someone in the company who presents a slide that shows an A/B test results that suggest that if you add one more ad, you get $100B in revenue with no negative impact. Everyone is excited and nobody ends up asking if a new user will like that experience. If someone asks, the answer is that “new users are only 0.1% of our user base and we couldn’t even track their performance in an A/B test”. I find this perspective very shortsighted because the cumulative impact of that type of decision-making ends up bringing the company’s growth to zero within 2–4 years.</p><p><strong>Solution</strong></p><p>Always track new user adoption and experience in experiments, even if the segment of those users looks too small for anyone to care. Moreover, given that the nature of most A/B tests is to add features, try to reserve a portion of your tests for removing stuff and decluttering your product. One product that has done an exceptionally good job in this area is Notion, and maybe more of us should follow the philosophy of targeting intuitive design that feels natural but also allows for a range of experiences customizable by the user.</p><h3>There is too much snacking</h3><p>Let’s assume you’re getting really good at A/B testing. Maybe you’ve become part of the tier of companies that run 1000 A/B tests a day. What you’ll find is that most of those A/B tests aren’t really testing for anything useful and represent small configuration changes.</p><p>I personally like these kinds of tests because they represent good learning opportunities for what should be the optimal decision boundary in a system. If done smartly, they can have a very good ROI on work. Unfortunately, because you’ve made A/B testing so easy, everyone within your company or team is always just “<a href="https://hunterwalk.com/2016/06/18/the-best-startups-resists-snacks-im-not-talking-about-food/">snacking</a>,” addicted to quick wins in the same way you get addicted to junk food. You’ll find that folks constantly move one number in a config to another until they get a positive outcome, but no real work happens. How do you get out of that local maximum?</p><p><strong>Solution</strong></p><p>I don’t know what the optimal split between real work and “snacking” is. Because snacking can also provide extremely attractive wins, you don’t want to completely discourage this behavior. Nonetheless, if you fundamentally don’t see any major product development from a team and all you see is “snacking”, it might be useful to realign incentives where the team is explicitly asked to take on more meaningful long-term work. Moreover, I recommend delegating snacking projects to newer or more junior members of the team, since it provides them with better learning opportunities. If you have a bunch of Staff+ engineers who only deliver impact through snacking, your incentives are probably not aligned well, since they are not working on solving the more challenging problems.</p><h3>Takeaways</h3><p>I generally find that if you do A/B testing long enough, you can get disillusioned about the overall process. Even with all the supposed rigor and objectivity, humans are ultimately involved in decision-making, and they’ll figure out a way to bring subjectivity back. Overall, if you want to see value in experimentation, I suggest focusing on the following:</p><ul><li>Setting a pre-registration program to minimize “everything looks significant” pathology. The more you deviate from the original experimental design, the less trust we should put in the outputs. Required pre-registration can help alleviate the issue of misaligned incentives.</li><li>Improving experimentation velocity. A good mandate would be to run most experiments in under two weeks. Exceptions should obviously exist, but the objective should be to get to a decision fast and minimize opportunity costs. This will also reduce the risk of misaligned incentives since folks are given more opportunities to try out ideas.</li><li>Pay attention to new user metrics. Even if those metrics are directional and have higher variance, simply by tracking them you might find that you’re doing too much harm to new user adoption because existing users have too much weight in your analysis.</li><li>Snacking should not be eliminated. It should be balanced against real work. I personally encourage for snaking to be done by new team members or junior members, since it presents new learning opportunities.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5a37cc252d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical testing: a guide for software engineers]]></title>
            <link>https://stassajin.medium.com/practical-testing-a-guide-for-software-engineers-ba4090bb5b4f?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba4090bb5b4f</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[integration-testing]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 18:35:30 GMT</pubDate>
            <atom:updated>2025-07-22T00:28:46.255Z</atom:updated>
            <content:encoded><![CDATA[<p>Testing is an essential part of the software development process, but it can be tedious, time-consuming, and might not prevent you from making mistakes. Moreover, code that is found in test directories is one of the highest liability code since it is less often scrutinized, highly duplicative, less documented, often ignored from linting rules, verbose, missing the code to regenerate fixture files, testing the wrong things, and is just overall harder to understand than the actual non-test code.</p><p>Unfortunately, a lot of the advice you might hear about testing practices is dogmatic and unjustified. I remember Uncle Bob once mentioning that test-driven development (TDD) is a done deal and you’re either incompetent or ignorant if you’re not using it. Yet I don’t enjoy TDD and I found that it’s rarely used in practice among many of my coworkers. Software engineering evolves in the same way things evolve in science. People come up with new processes and ideas for how to do things only to find that five years later the new generation has moved on to new paradigms, frameworks, and languages because those sometimes work better. Therefore it is best to keep an open mind and constantly question any advice, including the one in this post. Below I want to share some practical things that make testing more efficient and less of a bottleneck.</p><h3><strong>Test for realism</strong></h3><p>I used to rely heavily on mocks and faker libraries. How else would you run an integration test against a database without those? How else would you run tests in isolation without mocking some function call? The problem with mocking and faking is that you’re never really testing against a realistic environment. Yes, your code is isolated and runs fast, and you end up following Mike Cohn’s test pyramid, with a lot of unit tests, but it’s all meaningless if those unit tests don’t check for realistic behavior. I think the preference for unit testing with mocking is primarily due to computers being slower in the past and codebases being much larger, so the speed of testing was a relevant factor. Nonetheless, today computers are much faster, tools are better, and I don’t see a reason to not make integration and e2e tests be a larger share of the overall testing suite. Unrealistic unit tests, even if they are fast, create more entropy and cognitive load, catch fewer bugs, generate a false sense of security, and lead you to spend more time maintaining the testing suite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/0*Rh_NduDY9XR85B11.png" /><figcaption>The testing pyramid. <a href="https://martinfowler.com/articles/practical-test-pyramid.html">Source</a>.</figcaption></figure><p>For example, don’t mock your Postgres. Instead, add an integration in your docker-compose that uses a real database that runs locally or in CI/CD and connects to your service. Then have some scripts handy that allows you to populate the database with data similar to what is in production. Similarly, you could use tools like <a href="https://garden.io/">garden</a>.io to mimic complex production environments that run on k8s or rely on port-forwarding to a staging environment for integration testing against other internal services.</p><h3><strong>What are the consequences of bugs or mistakes?</strong></h3><p>Different industries have different requirements for how much coverage and how thorough your testing should be. I had the privilege of working for a financial institution that is heavily regulated and where mistakes could have severe consequences. I was very paranoid when working in that environment. We had multiple layers of high-fidelity testing with high coverage across unit, integration, e2e, and smoke tests. We were integrating MLOps practices of versioning data, ML models, documentation, and features before MLOps was a word or any vendor or open source tools in that area existed. We even adopted append-only storage for our tables so that we never, ever mutated or deleted data. That slowed us down, but that was overall a good thing because mistakes in that context were expensive to fix. I want the same level of care and paranoia to be present in engineers who handle my money, develop my medicine, and build my car.</p><p>Today I work for a last-mile logistics company where a mistake means that someone might get their food delivered late or not at all. If I brought that level of caution to my existing employer, I would generate higher opportunity costs than benefits and should be fired, since I would not able to ship product features with high velocity. Yes, mistakes will happen, but they would be far offset by the business impact of constantly improving the product. I’m ok with breaking something one out of ten times if there are processes in place that allow me to identify and mitigate the issue quickly. In this environment, testing is still relevant, but it should not consume the bulk of development time.</p><p>Try to understand if you operate in a low or high-risk environment and use that to guide the amount of time you spend on testing.</p><h3><strong>Prefer e2e tests first and unit tests last</strong></h3><p>In the past, I think folks were a lot more thoughtful about how they wrote their software. Compute was expensive and you were generally encouraged to get your code right on the first try. Modern IDEs and tooling have atrophied that skill, with most programmers often rewriting and refactoring the same code dozens of times until they think it looks good. Moreover, programming is a lot more collaborative today, with most of the code getting reviewed by multiple people who might have different opinions about how something should work. You can’t just push code unless it gets some eyes on it.</p><p>Because our habits of refactoring and writing code are so different from 20 years ago, a premature commitment to unit tests means that when you perform your rewrites, you’re also having to update the unit tests. Instead, my recommendation would be to frontload e2e tests and leave unit tests for last.</p><ul><li>Yes, I’m the person that opens PRs without tests because I want to get early feedback on my implementation before I waste time writing tests.</li><li>Yes, I’m also the person who prefers to test my public methods and leave all the helper functions and private methods for last.</li><li>And yes, I would sometimes not be willing to add tests and push code under a feature flag. After two or three rewrites, where I’m happy with how the code looks, I would then follow up with tests. Finally, I would turn the flag on.</li></ul><h3><strong>Rely on regular builds to achieve reproducibility</strong></h3><p>What is wrong with this Dockerfile?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*-j2ERZw3uMeH8c9Oj57XBA.png" /></figure><p>There are only three lines of code but so many things will go wrong. Rebuilding Dockerfiles does not guarantee reproducible environments. In our example, we pull `latest` image for ubuntu, leading to unpredictable operating system dependencies. Moreover, when we install git, how do you know which version will get installed? Recently, at work, we had a deploy bug caused because of an unexpected git dependency change. You just can’t trust anything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*igFyCM8AoUedDwyz" /></figure><p>Unfortunately, it is very hard to guarantee reproducible environments. This is particularly problematic in the Python ecosystem, especially if you don’t use tools like Poetry. You come back 6 months later to update your library and find that nothing works because of some transitive dependency changes.</p><p>Nightly builds help alleviate this issue. On a daily cadence, your CI kicks off an operation that builds your software artifact and runs the test suite against it. You can do this for both libraries and services. If the artifact fails to build, 99% of the time is because there was an unforeseen dependency drift that you should fix. At least now you know about something breaking ahead of time rather than being put in a position to scramble an emergency security patch in a code artifact that completely fails to build and deploy.</p><h3><strong>Property-based testing is overrated</strong></h3><p>If you don’t know what property-based testing is, that’s good. Skip this section and don’t look back. If you’re a huge fan, I’m unlikely to convince you of my arguments since you&#39;re probably putting far too much weight on functional programming and type theory.</p><p>My litmus test is that if a paradigm or framework does not bring me joy, I should move on. Property-based testing is one of those paradigms that I dislike and would use only if the consequence of mistakes in code is high. I wrote hundreds of property-based tests, and I just can’t stomach them anymore. For example, a lot of code I write is mathy in nature and I don’t want to just test the property that when I add two positive numbers A and B, the result C should be ≥ B,A. I want instead to test that when I add two numbers I get exactly the right result. Property tests take longer to write, run slower, are not always deterministic, and most importantly are just much harder to read and reason about than simple unit tests.</p><p>Property testing is not completely invaluable. For some critical libraries or applications, for high-risk domains, they can be useful in catching bugs. Nonetheless, in product development, their ROI is marginal relative to simple unit tests.</p><h3><strong>Combine documentation and tests</strong></h3><p>As I mentioned previously, one reason why tests are high liability code is that it is often less scrutinized than regular code. One way to perform better testing is to use <a href="https://www-cs-faculty.stanford.edu/~knuth/lp.html#:~:text=Literate%20programming%20is%20a%20methodology,in%20a%20high%2Dlevel%20language.">literate programming</a>. In literate programming, you combine the programming language with a documentation language. For example, consider the documentation for <a href="https://docs.pydantic.dev/usage/models/">pydantic</a> or <a href="https://docs.fast.ai/">fast.ai</a>. The actual code exposed in documentation gets run whenever the documentation artifact is built and the result of that is that you have both tests and documentation all in one place. This practice elevates just how much attention you pay to your testing code and encourages a more thoughtful frame of mind for how your write tests since you&#39;re not simply testing your code, you’re proudly showcasing how it works.</p><h3><strong>Use A/B testing as a testing paradigm</strong></h3><p>Only after I joined a larger company I learned the value of using experimentation as a testing paradigm. At large companies where a single team runs hundreds of experiments, effective teams work in something similar to 1–2 day sprints. They brainstorm an idea internally one day, push the code to a subset of users the following day, and then monitor for metric impact. If you’re obviously pushing something wrong to users, this will show in your observability and A/B test metrics.</p><p>In an experimentation-driven workflow, the value of tests decreases because a lot of changes you make will be reverted if their business impact is not justified. In this environment, do you want to spend more time writing comprehensive testing for features that might not even be fully shipped? Would you not rather ship 2x or 3x times the number of features if you can partially skip some tests?</p><p>Note that running A/B tests doesn’t obviate the need to run any tests. You still write them, but the coverage is less comprehensive.</p><h3><strong>Add more tests as iteration slows down or code matures</strong></h3><p>After you reach some maturity in your code, it is preferable to go back and add additional tests. Tests have two purposes: 1) to check for code correctness and 2) to prevent accidental regressions or bugs when you make code changes. When you iterate daily on a piece of code, you’re much more confident about its correctness because you’re probably testing/using the code a lot. Nonetheless, come back to your code six months later, and you’ll find that you don’t remember everything. The purpose of tests is to prevent yourself or new code owners from shooting themselves in the foot due to a lack of familiarity.</p><h3><strong>Lean on health checks</strong></h3><p>I think SRE and DevProd engineers have done a great job in introducing health checks that monitor the availability of a service. These health checks are used to perform automatic failover and recovery, load balancing, or optimize resource usage. Nonetheless, one area that I think is being underinvested is having health checks that continuously run tests in a production environment and proactively send alerts on those failures to product teams. The tests themselves are nothing more than “golden path” e2e tests that get triggered on a cadence and any data associated with those tests gets scaffolded and purged at each run.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba4090bb5b4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why process stifles innovation]]></title>
            <link>https://medium.com/swlh/why-process-stifles-innovation-d8622b7e8cae?source=rss-d9a8a91ee988------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8622b7e8cae</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[work]]></category>
            <category><![CDATA[enterpreneurship]]></category>
            <dc:creator><![CDATA[Stas Sajin]]></dc:creator>
            <pubDate>Thu, 24 Nov 2022 19:06:07 GMT</pubDate>
            <atom:updated>2024-03-06T20:58:25.088Z</atom:updated>
            <content:encoded><![CDATA[<p>Lately, I kept reflecting on two topics that share a similar theme. The first topic has to do with the state of California’s public infrastructure and how inefficient it has become over the years. The second is the failure of large tech companies to innovate and execute more rapidly than startup incumbents. Let me provide some perspectives on why these topics are related by first giving you two examples to think about:</p><ul><li><strong>Infrastructure</strong>: The Golden Gate Bridge took about four years to build (1933–1937) and it cost us ~$500M in today’s dollars. By contrast, the SDS (<a href="https://www.goldengatebridgenet.org/">Suicide Deterrent System</a>), which basically adds a net around the bridge, will take us five years to complete (2017–2023) and is projected to cost us ~$217M. If you do a review of major infrastructure projects in California, you’ll find that for projects completed before the 1950s the work was done faster, cheaper, and with higher quality than it is done today by at least a 5x magnitude, even though today we have more capital, better technology, better raw materials, and a higher absolute number of civil engineers and architects.</li><li><strong>Tech Companies</strong>: Adobe, despite all its financial and human resources, finds itself having to acquire a competitor for $20B. Whether Adobe overpaid for Figma remains to be seen, but the more interesting question is why did a company with 100x the personnel, 100x the depth of experience in design tooling, and 100x the financial capital fail to win?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nmnhTmFd3j_Pevlx" /><figcaption>Photo by <a href="https://unsplash.com/@sanathk?utm_source=medium&amp;utm_medium=referral">Sanath Kumar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In both cases, something went wrong, and it is interesting to examine why. Digging into the infrastructure issue, I stumbled on an interesting <a href="https://www.liberalcurrents.com/the-case-for-abolishing-the-national-environmental-policy-act/">explanation</a> that a big problem has to do with the National Environmental Policy Act (NEPA), which although well-intentioned, has created a hyper-restrictive review process that often blocks projects that are meant to improve our impact on the environment. Under current regulations, projects like the Golden Gate Bridge are basically impossible to get through.</p><p>Similarly, if you take a look within companies, you might find that as they grow, process and bloat creeps into anything and stifles real work and innovation. In that environment, the employees that are the most eager to create things, the makers, find that it is just more attractive to jump ship to a smaller place where their work does have an outlet to materialize. Both infrastructure failures and company innovation failures are at least partially due to bloat and process creep.</p><h3>Why does process creep happen?</h3><p>Every process starts with good intentions.</p><ul><li>“We want to make sure all stakeholders have an input into the final work, so all submitted project proposals need to be reviewed by legal, design, product, and engineering teams.”</li><li>“We want to make sure that large infrastructure projects don’t cause harm, so we need to perform thorough environmental studies before any work is started.”</li><li>“We want to minimize data breaches, so any request for data needs to be done through the IT and Security teams.”</li></ul><p>At the outset, these are innocuous and sensible suggestions. No process by itself is a catalyst that prevents work from being done, and some processes improve efficiency and prevent mistakes, yet if you take the cumulative impact of all the rules and requirements within an organization, you find that they ultimately have a cost on execution. Below I have a list of reasons why I think process creep happens and possible resolutions.</p><h4><strong>The road to perfectionism</strong></h4><p>I remember hearing a well-intentioned manager tell his direct reports to write software as if it would last 100 years. Although the remark was hyperbolic, the advice echoes something that I’ve heard often in the tech industry through adages like “measure twice and write once” or “if you don’t have time to do it right, do you have time to do it twice?.” This is not bad advice. In some situations, you have to write code as if it will last <a href="https://www.youtube.com/watch?v=H62hZJVqs2o&amp;ab_channel=StrangeLoopConference">100 years</a>, and going headfirst into execution without planning is wasteful. Nonetheless, extreme emphasis on planning can lead to paralysis and failure to recognize the benefits of learning through iteration. If your organization relies on planning everything to perfection before a single line of code is written, you might have a hard time moving in the right direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EpQgYTnRNLwMDu_V.jpg" /></figure><h4>Too much glue</h4><p>Many years ago I read Tanya Reilly’s blog post on <a href="https://noidea.dog/glue">being Glue</a> (highly recommended read, btw). At that point in my career, the content really resonated because I was dealing with a few colleagues who although technically competent, were just not working on the right things. Glue people engage in a process of technical leadership and mentorship that leads to making other people on the team more productive. Some amount of glue is necessary, nonetheless, I found that too much glue creates a set of other problems:</p><ul><li>Folks doing glue work prevent other people from improving their non-technical skills. For example, a Tech Lead who does not let their team speak about their work or often acts as a “buffer” is essentially taking opportunities away from ICs to struggle and eventually improve their communication skills.</li><li>Extreme and toxic versions of glue behavior sometimes manifest in taking credit for other people’s work or in generating grossly exaggerated attribution (“e.g., without me acting as an intermediary, the project would not succeed”). While Tanya talks about the situations where Glue people often don’t get acknowledged, my experience is completely the opposite. Because Glue people are good communicators, they know how to advertise their work and play politics well. In return, they end up with higher visibility and more promotion opportunities, while the engineer who works on implementation details often gets token acknowledgment or no credit at all.</li><li>Lastly, adding too many glue people reduces ownership and accountability for individual engineers. Instead of adding glue people to manage a technically competent team, I would simply let go of engineers that seem to have no business intuition, lack communication skills, or can’t put a paragraph together describing why the work they do is impactful. Suggesting that you need more glue to manage engineers ends up dismissing the general pattern that folks that are technically brilliant also have a high level of general intelligence, read a lot, are lifelong learners, and are very good at business and not just code.</li></ul><p>Note that I’m not alone in the opinions above. Here is what Eric Schmidt, former CEO of Google, has to say about glue roles.</p><blockquote><em>When I was at Novell, I had learned that there were people who I call “glue people”. The glue people are incredibly nice people who sit at interstitial boundaries between groups, and they assist in activity. And they are very, very loyal, and people love them, and</em><strong><em> </em></strong><em>you don’t need them at all.</em></blockquote><blockquote><em>At Novell, I kept trying to get rid of these glue people, because they were getting in the way, because they slowed everything down. And every time I get rid of them in one group, they’d show up in another group, and they’d transfer, and get rehired and all that.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhcRxFRgNpns%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhcRxFRgNpns&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhcRxFRgNpns%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f068ddce839f92f2bae17229e24697fe/href">https://medium.com/media/f068ddce839f92f2bae17229e24697fe/href</a></iframe><h4>Entropy</h4><p>As organizations grow, institutional knowledge, the number of stakeholders, codebases, and entropy increase. Increased entropy leads to high waste, high cognitive load, disorganization, and chaos.</p><p>As software engineers, I think we’ve done a crappy job in managing complexity. For example, for years we’ve heard that monoliths are bad because they generate bloated codebases, make deployments difficult, or create issues with scalability. Yet on the other extreme, where we have to operate in a world of microservices, I find that in order to ship basic things I need to operate across multiple codebases written in different languages, each having their deployment quirks and architecture patterns, with the whole paradigm turning into <a href="https://medium.com/codex/managing-serverless-spaghetti-c2fb1a7db73e">lambda spaghetti</a>. Stopping and refocusing on what&#39;s important has a big impact on reducing entropy. At a low, action-oriented level it can manifest as follows:</p><ul><li>Deleting dead code that is not executed in production. I’ve interacted with so many codebases where &gt;50% of the code was dead code and created cognitive load for colleagues. I wish there was something in the dev productivity space that generated code coverage based on lines executed in production. Every quarter you stop and review those reports and simply delete files that had zero execution.</li><li>As product managers, we don’t reflect enough on what features should we kill or revert. Yet stopping and reflecting on what should be deleted can be very meaningful in simplifying and improving user experience. I consider myself tech-savvy, yet I find that many of the products created today have been hyper-optimized on a set of existing users, leading to a cluttered and confusing UX experience for new users. Asking at the end of the quarter “What feature can we unship?” can be useful in reducing clutter.</li></ul><p>Lastly, I think a big reason for increased entropy is the failure of companies to retain top talent. Again and again I find that there is just no incentive for many folks to stay at a company beyond 2–3 years, even if they are a top performer. Any human capital loss means that institutional knowledge gets lost and new replacements will often engage in an exercise of reinventing the wheel without making any meaningful improvements. If you want reduced entropy, push the same set of people to rapidly iterate on the product for a sustained period of time.</p><h4><a href="https://www.amazon.com/Ego-Enemy-Ryan-Holiday/dp/1591847818">Ego is the enemy</a></h4><p>Ryan Holiday suggests in his book Ego is the Enemy, there is a strong “temptation that exists for everyone — for talk and hype to replace action.” Ego is a big reason why progress and innovation can be lackluster in many institutions. Here is how ego manifests itself:</p><ul><li>Seeing the same people itching to pitch their thoughts into a group conversation, even though their ideas add no or very little marginal value.</li><li>Having people that never engage in actively sponsoring their colleagues or who try to hoard visibility. If you’re a manager or a manager of managers, you can notice this by seeing how often your direct report speaks about other people’s successes or contributions in your 1:1s. Lack of acknowledgment or selfless sponsorship of other folks can be a sign of an unhealthy ego.</li><li>Optimizing for the team instead of the overall company or org. I borrowed the diagram from the <a href="https://www.oreilly.com/library/view/the-staff-engineers/9781098118723/">Staff Engineer’s Path</a> by Tanya Reilly and it perfectly exemplifies this problem where teams optimize for the local maximum. For example, would you be ok with disbanding your team and redistributing employees to other teams if within your domain you’re entering maintenance mode? Most managers would never do that because it would mean losing any social and political capital within the company.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*pe1Zrjvd1UXwO7Lw-vtMsA.png" /><figcaption><a href="https://www.oreilly.com/library/view/the-staff-engineers/9781098118723/">Source</a></figcaption></figure><ul><li>Depending on your role, there is a very strong dichotomy in terms of how you view the value of meetings. “<a href="http://www.paulgraham.com/makersschedule.html">Most powerful people are on the manager’s schedule</a>”. Unfortunately, their perception of the value of meetings creates a cultural override that affects people who prefer to have uninterrupted time to think about a problem and solution. Even for brainstorming purposes, the evidence suggests that <a href="https://hbr.org/2015/03/why-group-brainstorming-is-a-waste-of-time">group brainstorming is a waste of time</a>. Yet why do so many bloated organizations end up with half of the ICs’ time being spent in meetings?</li></ul><p>Overall I don’t think there are sensible solutions to the Ego problem. This is a lot more challenging to fix and often requires having the right people at the top leading through example.</p><h3>Conclusion</h3><p>The engine of innovation relies on evolutionary change. Ultimately, if an institution becomes rigid and does not adapt to change, it will die a slow death. In my experience, introducing a process often has unintended costs that lead to reduced velocity. Instead, I would focus on the following:</p><ul><li>Favoring iteration and learning from mistakes instead of trying to achieve perfectionism.</li><li>Glue people are necessary, but there has to be a limit to their number, so that they can act with high leverage. Simply hiring folks that are both technically competent and have good business intuition can solve the need of overhiring for glue roles.</li><li>Lastly I would focus on reducing entropy by constantly asking what can be discarded and rewarding folks for behavior that increases simplicity.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8622b7e8cae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/why-process-stifles-innovation-d8622b7e8cae">Why process stifles innovation</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>