<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dina Bavli on Medium]]></title>
        <description><![CDATA[Stories by Dina Bavli on Medium]]></description>
        <link>https://medium.com/@dinabavli?source=rss-ae5cca1dfa71------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qEoYM1MAVtj2mnpTvPydrg.jpeg</url>
            <title>Stories by Dina Bavli on Medium</title>
            <link>https://medium.com/@dinabavli?source=rss-ae5cca1dfa71------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:28:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dinabavli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Digital Experiments: The Statistics Were Never the Hard Part]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-the-statistics-were-never-the-hard-part-bd77a7ef8efe?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd77a7ef8efe</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 06:11:01 GMT</pubDate>
            <atom:updated>2026-11-06T06:11:01.673Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Eleven parts later, the durable lesson is not a formula. It is how to make a defensible decision before, during and after an experiment.</em></h4><p><em>Part 12 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQVaO2SJgUGy-F-GxLAREw.png" /></figure><p>After eleven parts, one pattern is hard to ignore.</p><p>Many of the most expensive experiment failures happen before anyone computes a p-value.</p><p>That is not an argument against statistics. The statistics matter enormously.</p><p>But they arrive after a sequence of upstream decisions that already determine what number you will get, what that number means, and whether acting on it is justified.</p><p>Should the experiment run at all?</p><p>What exactly is the treatment?</p><p>What is the outcome supposed to represent?</p><p>Who is randomized, and who else can be affected?</p><p>How much evidence is enough?</p><p>And if the result wins, do you trust the experiment that produced it?</p><p>That was the argument of the series.</p><h3>The calculation sits downstream of the design</h3><p>The first part began before the test, with the question of whether the decision even deserved an experiment.</p><p>Then came confounding: a larger dataset can make a biased answer look more precise without making it more correct.</p><p>Potential outcomes forced the uncomfortable point that the causal quantity we want is partly missing by construction.</p><p>Metrics moved the argument into the business itself. The OEC was never merely a dashboard field. It encoded what the company values and what it refuses to damage.</p><p>Then the series moved into design.</p><p>Sometimes randomization is available. Sometimes it is not. Sometimes the right design is a field experiment, a switchback, a quasi-experiment, or no defensible design at all.</p><p>The randomization unit matters because treatment can leak through shared resources, markets, feeds, households, conversations or model context.</p><p>The treatment definition matters because labels such as “personalization” can hide radically different interventions.</p><p>Only after those choices do power, sample size and duration become meaningful questions.</p><p>Even then, “how much evidence is enough?” is not answered by 0.05. It depends on the effect worth acting on, the cost of each kind of error, how reversible the decision is, and how long you are willing to expose users to uncertainty.</p><p>Then come the trust checks.</p><p>A clean p-value cannot rescue a broken assignment pipeline, post-treatment filtering, differential logging or instrumentation drift.</p><p>And after all of that, a statistically real effect can still be the wrong thing to ship.</p><p>The experiment answers a causal question. The organization still has to make a decision.</p><h3>AI changed the object, not the discipline</h3><p>The eleventh part brought the same framework into generative systems.</p><p>This is where it is especially easy to overstate what has changed.</p><p>Non-determinism does not automatically make a treatment ill-defined. A stochastic policy can be a perfectly coherent intervention if the model version, prompt, tools, retrieval, routing and relevant configuration are specified.</p><p>The real risks are hidden versions of treatment, unstable measurement and drift.</p><p>An LLM judge is still a measurement instrument.</p><p>A model update during the run can still create an instrumentation problem.</p><p>A synthetic user is still evidence whose validity has to be established for the question you care about.</p><p>The object is new.</p><p>The reasoning is not.</p><h3>The eight questions underneath every experiment</h3><p>If I had to compress the entire series into one reusable framework, it would be these eight questions.</p><ol><li><strong>Should you experiment at all?</strong><br>Will the answer change a real decision?</li><li><strong>What counts as evidence here?</strong><br>What is the strongest design your constraints actually allow, and what uncertainty remains?</li><li><strong>What is the OEC?</strong><br>What outcome decides the experiment, and what guardrails are allowed to veto it?</li><li><strong>What design does the constraint permit?</strong><br>Randomized trial, switchback, quasi-experiment, or no defensible design?</li><li><strong>What gets randomized?</strong><br>User, session, account, market, time block, conversation?</li><li><strong>What exactly is the treatment?</strong><br>Not the adjective on the slide. The intervention actually delivered.</li><li><strong>How much certainty is enough?</strong><br>MDE, power, alpha and duration tied to the decision, not inherited from convention.</li><li><strong>Do you trust the result?</strong><br>Assignment, SRM, instrumentation, validity, analysis choices and everything else that can turn the experiment you ran into a different experiment by the time it reaches the dashboard.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vQcCd_Ukcp4Zz9KnC3dX5w.png" /></figure><p>“It depends” is often the correct professional answer in experimentation.</p><p>But it is only useful if you can say <strong>what it depends on</strong>.</p><p>That is the real skill.</p><h3>What survives the tools</h3><p>Google Optimize is gone.</p><p>Experimentation platforms change.</p><p>Feature-flag stacks change.</p><p>Model providers change.</p><p>Evaluation harnesses that feel state of the art today will age quickly.</p><p>The durable questions are slower to expire.</p><p>Who received the intervention?</p><p>Who did not?</p><p>What exactly changed?</p><p>What outcome stands in for the thing you care about?</p><p>What assumption makes the comparison causal?</p><p>What could have contaminated it?</p><p>What would make you reverse the decision?</p><p>And what is the cost if you are wrong?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qbK1fTPbIVjm1bg6-BZ-Aw.png" /></figure><p>This is why the title of this final part is not “statistics do not matter.”</p><p>They do.</p><p>The point is narrower and more useful:</p><p><strong>A good experiment is not one with a good p-value. It is one where you can defend why the resulting decision is justified.</strong></p><p>Sometimes that decision is ship.</p><p>Sometimes it is hold.</p><p>Sometimes it is rerun with a better design.</p><p>Sometimes it is “we cannot answer this credibly with the data and constraints we have.”</p><p>And sometimes the best experiment is the one you decide not to run.</p><p>That is not a failure of experimentation.</p><p>That is experimentation doing its job.</p><h3>Questions to think about</h3><ul><li>In your last experiment, which upstream decision mattered more than the statistical test itself?</li><li>Which of the eight questions is weakest in a project you are running now?</li><li>What is the next experiment you should stop and rewrite before anyone splits traffic?</li></ul><p>If you want to revisit the mechanics rather than the argument, the <a href="https://github.com/dinbav/digital-experiments-course">notebooks that accompany the series</a> cover confounding, OECs, interference, power, CUPED, peeking, SRM, A/A tests, difference-in-differences and LLM evaluation.</p><h3>Sources and further reading</h3><p>This closing part synthesizes cases sourced in full in the earlier articles. The main primary sources referenced across the series include:</p><ul><li>Greg Linden, “Early Amazon: Shopping cart recommendations,” <em>Geeking with Greg</em>, 2006. <a href="https://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html">https://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html</a></li><li>Thomas Blake, Chris Nosko &amp; Steven Tadelis, “Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment,” <em>Econometrica</em> 83(1), 2015. <a href="https://doi.org/10.3982/ECTA12423">https://doi.org/10.3982/ECTA12423</a></li><li>Ravi Bapna, Jui Ramaprasad, Galit Shmueli &amp; Akhmed Umyarov, “One-Way Mirrors in Online Dating: A Randomized Field Experiment,” <em>Management Science</em> 62(11), 2016. <a href="https://doi.org/10.1287/mnsc.2015.2301">https://doi.org/10.1287/mnsc.2015.2301</a></li><li>Aleksander Fabijan et al., “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners,” <em>KDD</em>, 2019. <a href="https://doi.org/10.1145/3292500.3330722">https://doi.org/10.1145/3292500.3330722</a></li><li>Tom Blake, Sarah Moshary, Kane Sweeney &amp; Steve Tadelis, “Price Salience and Product Choice,” <em>Marketing Science</em> 40(4), 2021. <a href="https://doi.org/10.1287/mksc.2020.1261">https://doi.org/10.1287/mksc.2020.1261</a></li><li>US Federal Trade Commission, “FTC Secures Historic $2.5 Billion Settlement Against Amazon,” 25 September 2025. <a href="https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-secures-historic-25-billion-settlement-against-amazon">https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-secures-historic-25-billion-settlement-against-amazon</a></li><li>Ron Kohavi, Diane Tang &amp; Ya Xu, <em>Trustworthy Online Controlled Experiments</em>, Cambridge University Press, 2020.</li><li>Alan Gerber &amp; Donald Green, <em>Field Experiments</em>, W. W. Norton, 2012.</li></ul><h3>The series</h3><ol><li><a href="https://medium.com/@dinabavli/digital-experiments-should-we-even-run-an-experiment-db90e6a848a1">Should We Even Run an Experiment?</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-more-data-wont-fix-your-bias-2e59de7020d0">More Data Won’t Fix Your Bias</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-the-outcome-you-never-get-to-see-dcee2bca8f07">The Outcome You Never Get to See</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-your-metric-is-a-business-decision-17bdb126ac0a">Your Metric Is a Business Decision</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-not-every-experiment-is-an-a-b-test-458159b78e15">Not Every Experiment Is an A/B Test</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-your-control-group-isnt-a-control-group-df9e7155e9ac">Your Control Group Isn’t a Control Group</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-youre-not-testing-what-you-think-7f8c066bcf3e">You’re Not Testing What You Think</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-how-much-evidence-is-enough-8f4621c685d6">How Much Evidence Is Enough?</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-can-we-trust-this-result-04e9e2925073">Can We Trust This Result?</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-the-experiment-worked-it-cost-2-5-billion-119b57ab8b7b">The Experiment Worked. It Cost $2.5 Billion.</a></li><li><a href="https://medium.com/@dinabavli/digital-experiments-when-the-treatment-is-a-language-model-7dcc1561a5e6">When the Treatment Is a Language Model</a></li><li><strong>The Statistics Were Never the Hard Part</strong></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd77a7ef8efe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: When the Treatment Is a Language Model]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-when-the-treatment-is-a-language-model-7dcc1561a5e6?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dcc1561a5e6</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[causal-inference]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 06:11:01 GMT</pubDate>
            <atom:updated>2026-11-03T06:11:01.643Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>What non-determinism actually changes in an experiment — and what randomization, measurement and causal design still demand</em></h4><p><em>Part 11 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tp_SzfndRDH7DJ1YJ5-sOQ.png" /></figure><p>Same user. Same prompt. Same model version.</p><p>Run it twice and you may get two different answers.</p><p>So what, exactly, is the treatment?</p><p>That question makes experimentation with generative systems feel fundamentally different from ordinary A/B testing. If treatment delivery itself is stochastic, it is tempting to conclude that the old framework no longer fits.</p><p>Most of it still does.</p><p>Randomization still has to work. The assignment unit still has to match the mechanism. The outcome still has to represent a decision you care about. Guardrails still have to stop a harmful launch.</p><p>What changes is more specific: <strong>how the treatment is defined, how subjective output is measured, and how you know that the system tested on day one is still the system being tested on day ten.</strong></p><h3>The treatment is not one generated answer</h3><p>For a deterministic UI change, treatment is easy to describe.</p><p>Variant A shows one checkout. Variant B shows another.</p><p>With a generative model, “the user received response X” is usually the wrong level of definition. The same deployed system may produce X, Y or Z.</p><p>A more useful treatment definition is the <strong>policy or system that generates the distribution of responses</strong>:</p><ul><li>model and exact version</li><li>system prompt</li><li>generation settings</li><li>retrieval configuration</li><li>tools</li><li>routing</li><li>safety layer</li><li>relevant state</li></ul><p>Non-determinism alone does not make causal experimentation impossible. Stochastic interventions can be well-defined. The problem is hidden treatment variation: materially different versions of the intervention sharing the same label.</p><p>That is a familiar causal-inference problem. The consistency or well-defined-intervention assumption asks whether “treatment A” is specific enough that its relevant versions are not quietly producing different causal effects. Recent methodological work makes the same point directly: when the “how” of an intervention matters, the intervention needs to be specified more precisely. <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC12225681/">https://pmc.ncbi.nlm.nih.gov/articles/PMC12225681/</a></p><p>For generative systems, this changes the estimand itself. The useful question is often closer to:</p><blockquote><em>What is the expected effect of deploying policy B rather than policy A over the population and traffic distribution we care about?</em></blockquote><p>That sounds like a small wording change. It is not.</p><p>The object being compared is a stochastic system. Its behavior is therefore a distribution, not a single canonical answer. Two systems can have almost identical mean quality while differing sharply in variance, tail risk, refusal rate, hallucination frequency, cost, or latency. A product team that reports only the mean can easily declare a tie between systems that users experience very differently.</p><p>No user experiences the average response. They experience one draw from the deployed policy.</p><p>That is why distributional evaluation is not an optional embellishment here. It is part of the treatment definition and part of the decision rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzmjFDX4avmqN5mUpBAxrg.png" /></figure><h3>The assignment unit becomes part of the product architecture</h3><p>Conversational systems introduce several plausible units: user, session, conversation and turn.</p><p>They are not interchangeable.</p><p>Suppose every turn is randomized independently between two models. Turn 4 depends on the context created by turns 1 through 3. If those earlier turns came from a different treatment, one arm has changed the state in which the other arm operates.</p><p>That is interference inside a conversation.</p><p>In many conversational products, assignment at the conversation or user level will therefore be more defensible than turn-level assignment. Not because “AI requires user-level randomization,” but because the assignment unit has to preserve the causal structure of the interaction.</p><p>This is the same principle that appeared earlier in marketplaces and switchback tests. Shared state changed the surface, not the logic.</p><h3>“Quality” is not an OEC until someone defines it</h3><p>Checkout conversion is measurable.</p><p>“Answer quality” is not, until you decide what the construct contains.</p><p>Does a good response mean:</p><ul><li>correct?</li><li>helpful?</li><li>concise?</li><li>safe?</li><li>grounded in sources?</li><li>complete?</li><li>preferred by users?</li></ul><p>Those dimensions can conflict.</p><p>A longer answer may win preference while increasing latency and cost. A more conservative model may improve safety while reducing task completion. A creative answer may score well on style while losing factual accuracy.</p><p>The old lesson still applies: <strong>define the construct before the metric.</strong></p><p>And once the outcome becomes subjective, the measurement system becomes part of the experiment.</p><h3>LLM-as-judge is a measurement instrument</h3><p>An automated score can look objective because no human typed it.</p><p>It is still a judgment.</p><p>That puts LLM-as-judge inside an old framework: measurement reliability and validity.</p><p>Zheng et al.’s MT-Bench and Chatbot Arena work found that strong judge models achieved over 80% agreement with human preferences on their benchmark, around the level of agreement observed between human annotators there. The same paper documents position bias, verbosity bias and self-enhancement bias. <a href="https://arxiv.org/abs/2306.05685">https://arxiv.org/abs/2306.05685</a></p><p>Those are not contradictory findings.</p><p>LLM judges can be useful.</p><p>They still need calibration.</p><p>Before a judge becomes part of the OEC, I would want:</p><ul><li>agreement with humans on a relevant labelled sample</li><li>test-retest checks when the judging procedure is stochastic</li><li>position-bias checks for pairwise comparisons</li><li>tests for whether verbosity buys score without genuine quality</li><li>a frozen, recorded judge version and rubric</li></ul><p>If the judge changes during the experiment, the measuring instrument changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIwidxdOS4yv5_od2o8szw.png" /></figure><h3>Drift is different from randomness</h3><p>Consider a three-week experiment.</p><p>On day 9, the provider updates the model behind a stable alias.</p><p>The dashboard still calls the arm “Treatment B.”</p><p>The causal intervention may no longer be the same.</p><p>This is importantly different from ordinary generative randomness.</p><p>Non-determinism is variation <em>within</em> the defined policy.</p><p>Model drift is a change <em>to</em> the policy.</p><p>The closest classical validity threat is instrumentation: the tool used to measure or deliver the intervention changes during the study. In an AI system it can be worse, because the intervention itself may have changed while the experiment label stayed fixed.</p><p>There is no SRM-style statistical check that reliably tells you this happened.</p><p>The defense is operational:</p><ul><li>pin versions where possible</li><li>log the actual model ID on every request</li><li>version prompts, tools and retrieval configuration</li><li>record deployments on the experiment timeline</li><li>predefine what happens if the treatment changes mid-run</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7q4bAd7lrj7MfbYLGU1RQ.png" /></figure><h3>Synthetic respondents are useful evidence, but about a narrower question</h3><p>The idea is attractive: before spending real traffic, ask language models to simulate users.</p><p>There is credible evidence that this can work for some questions.</p><p>Argyle et al. reported <strong>algorithmic fidelity</strong> in “silicon samples”: conditioned GPT-3 responses reproduced meaningful patterns seen across human subgroups in several US survey datasets. <a href="https://doi.org/10.1017/pan.2023.2">https://doi.org/10.1017/pan.2023.2</a></p><p>But follow-up work exposes exactly the limitation an experimenter should care about.</p><p>Bisbee et al. found that synthetic survey responses had less variation than the corresponding human data, that regression coefficients could differ materially, and that response distributions changed with small prompt changes and across time. <a href="https://doi.org/10.1017/pan.2024.5">https://doi.org/10.1017/pan.2024.5</a></p><p>CoMPosT adds a different warning: simulated personas can become flattened or exaggerated caricatures of the groups they are meant to represent. <a href="https://doi.org/10.18653/v1/2023.emnlp-main.669">https://doi.org/10.18653/v1/2023.emnlp-main.669</a></p><p>The important distinction is between <em>simulation usefulness</em> and <em>causal substitutability</em>.</p><p>Synthetic respondents may be useful for exploration, stress testing, copy variants, rubric development, red-teaming or hypothesis generation. Those are all valuable jobs.</p><p>But when the target estimand is the effect of a real product treatment on actual people, the bar is higher. The simulator would need to preserve the response distribution that matters for that decision, including heterogeneity and tail behavior, not merely reproduce a few average patterns.</p><p>Under-dispersion is therefore especially dangerous in experimentation. If the simulated population is artificially tidy, uncertainty, tail behavior and heterogeneity all look easier than they really are. A synthetic population can make a weak intervention look more stable than it would be in production.</p><h3>What actually changed?</h3><p>Less than the tooling suggests, but enough that vague experiment specs become dangerous.</p><p>The classical pieces still matter: randomization, estimands, assignment units, power, guardrails, validity and pre-specification.</p><p>What changes is the granularity at which those ideas must be operationalized.</p><p>The treatment may be a stochastic policy rather than a single output. “Quality” needs an explicit construct and rubric. The judge becomes part of the measurement system. Cost and latency can stop behaving like secondary guardrails and become binding constraints. And versioning becomes causal metadata: without it, you may not be able to reconstruct the intervention that actually ran.</p><p>The durable lesson is simple:</p><p><strong>AI does not remove the principles of experimentation. It makes vague definitions more expensive.</strong></p><h3>What it depends on</h3><ul><li><strong>Treatment definition.</strong> Did you specify a system or merely write “LLM”?</li><li><strong>Assignment unit.</strong> User, conversation, session or turn?</li><li><strong>Judge.</strong> Who measures quality, and how well does that judge agree with humans?</li><li><strong>Versioning.</strong> Can you reconstruct which system every user actually received?</li><li><strong>Guardrails.</strong> Quality against latency, cost and safety.</li><li><strong>Synthetic evidence.</strong> Which exact question has the simulation been validated for?</li></ul><h3>Questions to think about</h3><ul><li>If the same system produces a different output on every run, what exactly is your treatment?</li><li>If an LLM is your judge, what human-labelled sample calibrates it?</li><li>If your provider changes model version mid-experiment, could you detect it from your logs?</li><li>What would you trust synthetic users to test, and what still requires real people?</li></ul><p>For readers who want to see the measurement problem numerically, <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/v2-ai-eval-demo.ipynb">this notebook</a> requires no API keys. It compares two judges on the same outputs, measures their agreement, checks whether scores track response length, shows a quality-versus-cost trade-off, and demonstrates how a mid-experiment quality drop can disappear inside an acceptable aggregate mean. <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/notebooks/v2-ai-eval-real-model.ipynb">The optional lab</a> runs related checks on real open-source models in Colab, and <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/notebooks/v2-ai-eval-exercise.ipynb">the blank version</a> leaves the analysis open.</p><h3>Sources and further reading</h3><ul><li>Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” <em>NeurIPS</em>, 2023 — human-preference agreement alongside position, verbosity and self-enhancement biases. <a href="https://arxiv.org/abs/2306.05685">https://arxiv.org/abs/2306.05685</a></li><li>Lisa Argyle et al., “Out of One, Many: Using Language Models to Simulate Human Samples,” <em>Political Analysis</em> 31(3), 2023, 337–351 — algorithmic fidelity and “silicon samples.” <a href="https://doi.org/10.1017/pan.2023.2">https://doi.org/10.1017/pan.2023.2</a></li><li>James Bisbee et al., “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” <em>Political Analysis</em> 32(4), 2024, 401–416 — under-dispersion, prompt sensitivity and reproducibility concerns. <a href="https://doi.org/10.1017/pan.2024.5">https://doi.org/10.1017/pan.2024.5</a></li><li>Myra Cheng, Tiziano Piccardi &amp; Diyi Yang, “CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations,” <em>EMNLP</em>, 2023. <a href="https://doi.org/10.18653/v1/2023.emnlp-main.669">https://doi.org/10.18653/v1/2023.emnlp-main.669</a></li><li>Ron Kohavi, Diane Tang &amp; Ya Xu, <em>Trustworthy Online Controlled Experiments</em>, Cambridge University Press, 2020 — treatment definition, interference, instrumentation and trustworthy experimentation.</li><li>Hall et al., “Inconsistent consistency: evaluating the well-defined intervention assumption in applied epidemiological research,” <em>International Journal of Epidemiology</em>, 2025 — why multiple treatment versions make precise intervention definition necessary. <a href="https://doi.org/10.1093/ije/dyaf015">https://doi.org/10.1093/ije/dyaf015</a></li><li>Course lecture: <a href="https://youtu.be/j9MEIQ3QKsk">Technologies for Intervention</a></li><li>Course lecture: <a href="https://youtu.be/mWk8IIA3IVE">Reliability</a></li><li>Course lecture: <a href="https://youtu.be/abi7CbPD698">Internal Validity</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dcc1561a5e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: The Experiment Worked. It Cost $2.5 Billion.]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-the-experiment-worked-it-cost-2-5-billion-119b57ab8b7b?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/119b57ab8b7b</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 06:06:01 GMT</pubDate>
            <atom:updated>2026-10-30T06:06:01.644Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>From statistical evidence to product decisions: economic significance, Simpson’s paradox, ITT, p-hacking, and the limits of optimization</em></h4><p><em>Part 10 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSbuY18yCVF89cM1OmFZ6A.png" /></figure><p>A checkout experiment ends with a +0.05% conversion lift at p = 0.001.</p><p>Ten million users saw the test. The result is highly significant.</p><p>Do you ship?</p><p>One missing fact changes the answer: the feature will cost $2 million to build and maintain.</p><p>The statistical question has been answered. The product question has not.</p><p>That gap is what this part of the series is about.</p><p>We spend enormous effort designing experiments that deserve to be trusted: randomization, power, assignment, treatment definition, validity, SRM checks. But once a clean result lands in a dashboard, teams often compress the rest of the decision into one binary label: significant or not significant.</p><p>A p-value cannot carry that weight.</p><h3>A real effect can still be the wrong decision</h3><p>Statistical significance asks how compatible the observed data are with a null model, under the assumptions of the test.</p><p>Economic significance asks something else: even if the effect is real, is it large enough to justify the cost, risk, and operational consequences of acting on it?</p><p>The distinction sounds obvious until the numbers arrive.</p><p>A primary metric rises 4% at p = 0.02, but a revenue guardrail falls 2%. That is not a statistics problem anymore. It is a decision about how the organization values engagement against money.</p><p>A checkout change moves conversion by 0.05%, with millions of users and a tiny p-value, but costs more to build than it is expected to return. The effect can be real and still be irrelevant.</p><p>The experiment did its job. The decision is yours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDRcXP-88RAKrgOcslU5LA.png" /></figure><p>There is a useful heuristic here, usually called <strong>Twyman’s law</strong>: an unusually interesting number deserves suspicion first. It is not a statistical theorem. It is a reminder to run sanity checks before turning surprise into narrative.</p><h3>The estimate depends on who you kept in it</h3><p>Suppose 1,000 users are randomized in an email-reminder experiment.</p><p>Five hundred are assigned to receive a reminder. Five hundred are assigned to control. Of the 500 assigned the reminder, only 200 actually open it.</p><p>Which group should you analyze?</p><p><strong>Intention-to-treat (ITT)</strong> keeps everyone in the arm they were assigned to. It preserves randomization and estimates the effect of <em>being offered</em> the treatment.</p><p>That is often the quantity the product team actually controls.</p><p>The tempting alternative is to compare the 200 openers with people who did not open, or with the whole control group, and call that “the treatment effect on the treated.”</p><p>It is not.</p><p>That is a naive per-protocol comparison. Opening the email was not randomized. People who open reminders are usually different from people who ignore them, even without the experiment. Once you condition on post-assignment behavior, the protection of randomization is gone.</p><p>There are principled ways to estimate effects among <strong>compliers</strong> using assignment as an instrument. Those estimands are useful, but they answer a different question and require additional assumptions.</p><p>The practical rule is simpler: before debating the number, name the estimand.</p><p>What are you trying to estimate: the effect of assignment, the effect of actual take-up, or the effect in a particular subgroup?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gEkc7oS56LCvXZcPR32yvg.png" /></figure><p>The same issue appears in a different form in <strong>Simpson’s paradox</strong>.</p><p>In UC Berkeley’s 1973 graduate admissions data, men had a higher admission rate overall than women. But department-level comparisons told a very different story because women had applied more often to departments that were more selective for everyone.</p><p>The aggregate was not numerically wrong. It mixed two mechanisms: where applicants applied and how departments admitted them.</p><p>That is why aggregation is a decision, not just a display choice.</p><p>But the reverse trap matters too. Slice enough subgroups and you will eventually find one that reverses the average. A pre-specified segment is evidence of one kind. A segment discovered after the result is visible is evidence of another.</p><h3>If you search long enough, something will cross 0.05</h3><p>The xkcd comic about green jelly beans causing acne is funny because it captures a very ordinary workflow.</p><p>Ask twenty questions. Nineteen are uninteresting. One returns p &lt; 0.05.</p><p>Report the one.</p><p>No fraud is required. Every individual test can be calculated correctly. The false-positive problem is created by the analysis process around them.</p><p>This is the operational form of p-hacking: flexible outcomes, flexible segments, flexible stopping, flexible reporting.</p><p>The fix is not a lecture about integrity. It is infrastructure and process.</p><p>Pre-specify the primary analysis. Keep an audit trail of what was actually tested. Report the relevant family of comparisons, not only the winner. Apply multiplicity corrections when the design calls for them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QdX6LjfHmB4kcSeDunhSDQ.png" /></figure><h3>Optimization can succeed and still fail the product</h3><p>A large field experiment on StubHub, published in <em>Marketing Science</em>, compared showing mandatory fees upfront with revealing them later in checkout.</p><p>Hiding the fees until late increased revenue. Showing them earlier reduced both purchase quantity and purchase quality. The researchers could trace part of the mechanism through clickstream behavior: shrouded prices made comparison harder, and users spent more than they otherwise would have.</p><p>If short-term revenue is the only objective, the optimization worked.</p><p>That is exactly why the example matters.</p><p>Statistics can tell you that the treatment caused the metric to move. It cannot decide whether the metric deserved to dominate the decision.</p><p>A much sharper version of that problem appears in the FTC’s case against Amazon over Prime enrollment and cancellation.</p><p>On 25 September 2025, the FTC announced a $2.5 billion settlement: a $1 billion civil penalty and $1.5 billion in refunds to an estimated 35 million consumers.</p><p>The public record does <strong>not</strong> establish that Amazon ran a controlled experiment with churn as the OEC and the “Iliad” cancellation flow as the treatment. That would be an invented story.</p><p>What the FTC alleged, citing internal Amazon documents, was that consumers were enrolled without clear consent, that cancellation was deliberately difficult, and that executives understood that making the experience clearer reduced subscriptions. The settlement imposed payments and forward-looking requirements; Amazon did not admit wrongdoing.</p><p>The experimentation lesson is not “Amazon ran a bad A/B test.”</p><p>It is that <strong>a metric can improve while the decision becomes worse</strong>.</p><p>If your objective function contains subscription retention but not consent, cancellation symmetry, trust, or regulatory exposure, no amount of statistical rigor will add those missing values for you.</p><h3>Some questions are outside the experiment’s optimization target</h3><p>Facebook’s 2012 “emotional contagion” experiment makes the boundary even clearer.</p><p>Researchers altered the News Feed of 689,003 users for a week, reducing positive content for some and negative content for others, then measured shifts in the emotional tone of users’ later posts.</p><p>The reported effects were tiny. The controversy was not mainly statistical.</p><p>It was about whether people should be subjected to an emotional manipulation without meaningful awareness or consent.</p><p>The Belmont Report and the Declaration of Helsinki come from formal human-subject research traditions. They do not automatically become the governing law for every private-sector product experiment. That mismatch is itself part of the problem.</p><p>A platform can ask, “Can we estimate this effect?”</p><p>The harder question is, “Should we create this treatment in order to estimate it?”</p><p>A p-value can contribute to the first answer.</p><p>It has nothing to say about the second.</p><h3>The experiment ends when the decision is explicit</h3><p>For a result to become a ship decision, I want at least five gates answered:</p><ol><li><strong>Can we trust the result?</strong></li><li><strong>Is the effect large enough to matter?</strong></li><li><strong>Are we looking at the right estimand and population?</strong></li><li><strong>Did the guardrails remain acceptable?</strong></li><li><strong>Is the treatment itself ethically and legally acceptable?</strong></li></ol><p>Only then does “ship” become a meaningful conclusion.</p><p>There is also a useful design-time test hidden in that list:</p><p><strong>“We would not run this experiment if ___.”</strong></p><p>The blank should contain a real boundary: a level of potential harm, a consent condition, a legal constraint, a type of manipulation, a risk to a protected population.</p><p>If the blank stays empty, the experiment has no declared ethical stop condition.</p><h3>What it depends on</h3><p>The decision changes with:</p><ul><li><strong>Effect size relative to implementation and maintenance cost</strong></li><li><strong>Metric conflicts</strong>, and who has authority to resolve them</li><li><strong>Pre-specified analysis</strong> versus post-result searching</li><li><strong>The estimand</strong> that actually corresponds to the product decision</li><li><strong>Regulatory context</strong>, which differs across jurisdictions and domains</li><li><strong>Participant harm</strong>, including harm created by a treatment that “works”</li></ul><h3>Questions to think about</h3><ul><li>If the p-value stayed identical but the implementation cost increased tenfold, would your decision change?</li><li>When you say “the treatment worked,” do you mean assignment, take-up, or a subgroup found after analysis?</li><li>Complete the sentence: “We would not run this experiment if ___.”</li></ul><p>For readers who want to explore this in code, <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/unit-10-analysis-demo.ipynb">this notebook</a> moves from inference to decision: a t-test and confidence interval on a simulated checkout lift, bootstrap inference for a skewed revenue metric, two definitions of the same ratio metric that disagree materially, and a statistically significant result that still falls below the ship threshold. It ends with a Simpson’s-paradox trap inside a ramp-up. The classic admissions table is in <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/unit-02-confounding-demo.ipynb">the confounding notebook</a>, and <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/notebooks/unit-10-analysis-exercise.ipynb">the blank version</a> leaves the analysis open.</p><h3>Sources and further reading</h3><ul><li>US Federal Trade Commission, “FTC Secures Historic $2.5 Billion Settlement Against Amazon,” 25 September 2025 — the $1B civil penalty, $1.5B in consumer refunds, and the settling parties. <a href="https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-secures-historic-25-billion-settlement-against-amazon">https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-secures-historic-25-billion-settlement-against-amazon</a></li><li>US Federal Trade Commission, <em>FTC v. Amazon.com, Inc.</em> (ROSCA) case page — the complaint, amended complaint, and settlement documents. Read the complaint as allegations. <a href="https://www.ftc.gov/legal-library/browse/cases-proceedings/2123050-amazoncom-inc-rosca-ftc-v">https://www.ftc.gov/legal-library/browse/cases-proceedings/2123050-amazoncom-inc-rosca-ftc-v</a></li><li>US Federal Trade Commission, “Bringing Dark Patterns to Light,” staff report, 2022. <a href="https://www.ftc.gov/reports/bringing-dark-patterns-light">https://www.ftc.gov/reports/bringing-dark-patterns-light</a></li><li>Tom Blake, Sarah Moshary, Kane Sweeney &amp; Steve Tadelis, “Price Salience and Product Choice,” <em>Marketing Science</em> 40(4), 2021, 619–636 — the StubHub field experiment. <a href="https://doi.org/10.1287/mksc.2020.1261">https://doi.org/10.1287/mksc.2020.1261</a></li><li>P. J. Bickel, E. A. Hammel &amp; J. W. O’Connell, “Sex Bias in Graduate Admissions: Data from Berkeley,” <em>Science</em> 187(4175), 1975, 398–404. <a href="https://doi.org/10.1126/science.187.4175.398">https://doi.org/10.1126/science.187.4175.398</a></li><li>Adam Kramer, Jamie Guillory &amp; Jeffrey Hancock, “Experimental Evidence of Massive-Scale Emotional Contagion Through Social Networks,” <em>PNAS</em> 111(24), 2014 — N = 689,003. <a href="https://doi.org/10.1073/pnas.1320040111">https://doi.org/10.1073/pnas.1320040111</a></li><li><em>PNAS</em>, “Editorial Expression of Concern: Experimental evidence of massive-scale emotional contagion through social networks,” 2014. <a href="https://doi.org/10.1073/pnas.1412469111">https://doi.org/10.1073/pnas.1412469111</a></li><li>Susan Fiske &amp; Robert Hauser, “Protecting Human Research Participants in the Age of Big Data,” <em>PNAS</em> 111(38), 2014. <a href="https://doi.org/10.1073/pnas.1414626111">https://doi.org/10.1073/pnas.1414626111</a></li><li>The National Commission, <em>The Belmont Report</em>, 1979. <a href="https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html">https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html</a></li><li>World Medical Association, <em>Declaration of Helsinki</em>, 2024 revision. <a href="https://www.wma.net/policies-post/wma-declaration-of-helsinki/">https://www.wma.net/policies-post/wma-declaration-of-helsinki/</a></li><li>Joshua Angrist, Guido Imbens &amp; Donald Rubin, “Identification of Causal Effects Using Instrumental Variables,” <em>JASA</em> 91(434), 1996. <a href="https://doi.org/10.1080/01621459.1996.10476902">https://doi.org/10.1080/01621459.1996.10476902</a></li><li>Randall Munroe, “Significant,” <em>xkcd</em> 882. <a href="https://xkcd.com/882/">https://xkcd.com/882/</a></li><li>Course lecture: <a href="https://youtu.be/Qk_j8KtAsUM">Business Interpretation</a></li><li>Course lecture: <a href="https://youtu.be/_DMWwm2K_BI">Simpson’s Paradox, ITT and ToT</a></li><li>Course lecture: <a href="https://youtu.be/BMmbADxBffA">Regression Analysis and P-Hacking</a></li><li>Course lecture: <a href="https://youtu.be/KC4MqXi_DLs">Ethics in Digital Experiments</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=119b57ab8b7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: Can We Trust This Result?]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-can-we-trust-this-result-04e9e2925073?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/04e9e2925073</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[product-analytics]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 08:01:03 GMT</pubDate>
            <atom:updated>2026-10-27T08:01:03.813Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>The strongest-looking result in the dashboard can still come from an experiment that quietly stopped being the one you designed</em></h4><p><em>Part 9 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxszSTTf7hzsY-mUJ4AYKQ.png" /></figure><p>Imagine the review meeting.</p><p>Millions of users.</p><p>A large effect.</p><p>A tiny p-value.</p><p>No obvious instrumentation errors.</p><p>Everyone is discussing rollout.</p><p>Then someone asks a question that is not on the result slide:</p><p><strong>How many users were actually observed in each arm?</strong></p><p>That question once changed the conclusion of a real experiment on Microsoft’s MSN portal. <br>The team was testing a homepage image carousel, increasing the number of rotating cards from 12 to 16.</p><p>Assignment worked.</p><p>Telemetry was collected.</p><p>The initial analysis said the new carousel hurt engagement.</p><p>But the experiment also failed a <strong>sample ratio mismatch</strong>, or SRM, check.</p><p>There were fewer observed users in one arm than the randomization plan could plausibly explain by chance.</p><p>The surprising part came from the root cause.</p><p>Treatment users interacted with the larger carousel more.</p><p>That extra engagement pushed some of them over the thresholds of a bot-detection system.</p><p>The filter removed them from analysis.</p><p>Real users disappeared from the dataset <strong>because the treatment had changed their behaviour</strong>.</p><p>Fix the filter, and the result flips.</p><p>The 16-card carousel was positive.</p><p>The experiment had not failed because randomization was broken.</p><p>It failed because the population being analysed was no longer the population randomization had created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6q2C9tYNgAgux9unDLrIA.png" /></figure><p>That is why trust checks belong <strong>before</strong> outcome interpretation.</p><h3>SRM is not the problem. It is the smoke alarm.</h3><p>A sample ratio mismatch occurs when the observed counts across variants differ from the configured allocation by more than sampling variation reasonably explains.</p><p>The usual test is simple.</p><p>For a 50/50 experiment, compare the observed counts with the counts implied by the assignment plan.</p><p>But detecting SRM is the easy part.</p><p>Explaining it is the work.</p><p>Microsoft’s taxonomy is useful because it follows the experiment from assignment to analysis.</p><p><strong>Assignment.</strong> Faulty IDs, incorrect bucketing, carryover.</p><p><strong>Execution.</strong> Redirects, variant-specific failures, different exposure or telemetry behaviour.</p><p><strong>Log processing.</strong> Joins, deduplication, event loss.</p><p><strong>Analysis.</strong> Filtering or segmentation based on something treatment can itself change.</p><p>The MSN bot detector lived in the last category.</p><p>The critical idea is that SRM does not say, “your randomizer is broken.”</p><p>It says:</p><p><strong>the data reaching analysis no longer match the experiment you planned.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wen7QMZhfsMMNgZHG-a4TQ.png" /></figure><h3>Why you should not rebalance your way out of it</h3><p>When teams first see an allocation mismatch, the temptation is operational:</p><p>Can we reweight?</p><p>Can we remove a few users?</p><p>Can we force the counts back toward 50/50?</p><p>That treats the warning as the defect.</p><p>The entire point of SRM is that you do not yet know <strong>why</strong> units are missing.</p><p>In the MSN case, missingness was created by treatment response.</p><p>The users who disappeared were not an interchangeable random subset.</p><p>They were among the users most affected by the new carousel.</p><p>Rebalancing the counts would have made the dashboard cleaner while preserving the bias.</p><p>The safe default is therefore much less satisfying:</p><p>stop outcome interpretation,</p><p>find the root cause,</p><p>repair it,</p><p>then decide whether the experiment can be recovered or needs to be rerun.</p><h3>Tiny ratio differences can be enormous warnings</h3><p>A 50.6/49.4 split does not look dramatic.</p><p>That is why visual inspection is the wrong test.</p><p>At a small sample size, a difference like that may be completely ordinary.</p><p>At millions of observations, the same proportional mismatch can be essentially impossible under the assignment plan.</p><p>SRM is therefore a statistical check on counts, not a requirement that dashboards display perfectly equal groups.</p><p>A mature experimentation system should run that check before teams get distracted by the outcome.</p><p>Microsoft explicitly describes SRM this way: as a trust condition that needs diagnosis before experiment effects are believed.</p><h3>A trustworthy p-value can answer the wrong experiment</h3><p>SRM belongs to a larger problem.</p><p>Teams often use the word <strong>valid</strong> to mean “the statistics were done correctly.”</p><p>That is only one layer.</p><p>A useful framework separates four questions.</p><h3>Construct validity</h3><p>Did the metric represent what you intended to measure?</p><p>A shorter checkout form may appear to win because completion rises.</p><p>But suppose the remaining promo-code field reminds users to leave the site and search for discounts.</p><p>The intervention affected behaviour through a mechanism that was not captured by the simple “fewer fields” story.</p><h3>Internal validity</h3><p>Did treatment actually cause the observed difference?</p><p>This is where selection, attrition, contamination and instrumentation enter.</p><p>Post-treatment filtering is an internal-validity threat because treatment helps determine who remains in the comparison.</p><h3>External validity</h3><p>Would the result survive outside the tested context?</p><p>Population matters.</p><p>Setting matters.</p><p>Time matters.</p><p>A reminder sent on Sunday morning and the same reminder sent during a weekday commute are not necessarily the same intervention in practice.</p><h3>Statistical conclusion validity</h3><p>Was the inference itself justified?</p><p>Sample size, variance, assumptions, peeking, multiple comparisons.</p><p>This is the layer most dashboards expose.</p><p>It is not the whole trust problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDkvP0l3NFPMJyt_Dgu-bQ.png" /></figure><h3>Three checks that answer different questions</h3><p>A good trust checklist is not one long list of green lights.</p><p>Each check protects a different part of the experiment.</p><h3>Allocation and SRM</h3><p>Did the observed population match the assignment plan?</p><p>If not, do not interpret the treatment effect until you understand why.</p><h3>Pre-treatment covariate balance</h3><p>Do characteristics measured before treatment look compatible with random assignment?</p><p>Perfect balance is not expected in every finite sample.</p><p>But systematic or implausible imbalances can reveal assignment, exposure or filtering problems.</p><h3>A/A tests</h3><p>Randomize users into two arms and give both arms the same experience.</p><p>Under a valid test at alpha 0.05, some A/A experiments should still produce statistically significant results by chance.</p><p>That is not a platform failure.</p><p>It is what the false-positive rate means.</p><p>This is why one clean A/A test proves very little.</p><p>The useful signal comes from repeated A/A experiments at scale.</p><p>Across many runs, the observed false-positive rate should behave like the method says it should. Microsoft has recently illustrated exactly this kind of platform-scale calibration using thousands of A/A scorecards.</p><h3>Some failures need a time axis</h3><p>Suppose allocation is correct.</p><p>There is no SRM.</p><p>The pre-treatment covariates look fine.</p><p>Treatment wins.</p><p>Now plot the effect by week.</p><p>Week one: large positive lift.</p><p>Week two: smaller.</p><p>Week three: barely there.</p><p>Week four: approximately zero.</p><p>The pooled estimate can still be positive.</p><p>But the experiment is now telling a different story.</p><p>A novelty effect that decays is not the same product as a durable behaviour change.</p><p>SRM cannot detect that.</p><p>Neither can a single end-of-test average.</p><p>You need the trajectory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ggcy7mo0bKPU_bZLQIrltA.png" /></figure><h3>The trust check that does not exist</h3><p>Now make the treatment a model.</p><p>The experiment starts on model version A.</p><p>Halfway through, the vendor updates the model behind the endpoint.</p><p>Nothing in assignment changes.</p><p>Nothing necessarily breaks in logging.</p><p>The dashboard continues to call the arm “treatment.”</p><p>But the arm now contains two interventions under one name.</p><p>There is no standard SRM-like test that reliably detects this after the fact.</p><p>The protection is operational:</p><p>pin versions where possible,</p><p>record them as part of the treatment definition,</p><p>log deployments on the experiment timeline,</p><p>and explicitly separate analysis when the intervention changes.</p><p>One of the most useful trust skills is knowing when there is no accepted automated check.</p><h3>Trust debt compounds</h3><p>A broken experiment is inconvenient.</p><p>A broken experimentation system is expensive.</p><p>Suppose the metric or pipeline in one experiment is wrong.</p><p>Then another experiment uses the same metric.</p><p>Then a third team ships based on it.</p><p>By the time someone finds the defect, the question is no longer whether one feature should be rolled back.</p><p>It is which previous decisions need to be reopened.</p><p>That is why experiment trust is not a final QA step.</p><p>It is infrastructure.</p><p>The order should be:</p><p><strong>verify the experiment, then interpret the effect.</strong></p><p>Not the other way around.</p><h3>What trust depends on</h3><p><strong>Assignment.</strong> Can you verify that units entered arms as designed?</p><p><strong>Observation.</strong> Can treatment affect whether a unit is seen at all?</p><p><strong>Pipeline.</strong> Redirects, event loss, joins, filters.</p><p><strong>Construct.</strong> Does the metric represent the thing the decision is about?</p><p><strong>Time.</strong> Is the effect stable, or is it novelty, primacy or drift?</p><p><strong>External validity.</strong> Does the result travel to the population, setting and period that matter?</p><p><strong>Instrumentation.</strong> Did the measurement system or treatment itself change during the run?</p><p><strong>Blast radius.</strong> What other decisions already depend on this metric or pipeline?</p><h3>Try the checks yourself</h3><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/unit-09-trust-checks-demo.ipynb">This notebook</a> turns three trust failures into simulations.</p><p>First, a logging bug manufactures a statistically significant win for a page that does nothing, while the SRM check reveals an observed split of 10,000 versus 7,600 instead of 50/50.</p><p>Then it runs 1,000 A/A experiments with zero true treatment effect so you can see why a healthy platform still produces approximately alpha-level false positives across repeated runs.</p><p>Finally, it plots a large week-one treatment effect that largely disappears by week four, showing what a pooled average can conceal.</p><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/notebooks/unit-09-trust-checks-exercise.ipynb">The blank version</a> contains the same setup with the analysis left open.</p><h3>Questions to think about</h3><ul><li>If SRM fails tomorrow but the treatment lift is excellent, do you still ship?</li><li>Can treatment change whether a unit is observed at all?</li><li>Which filters in your pipeline are applied after exposure and could therefore become post-treatment filters?</li><li>Do you inspect effect trajectories over time, or only the final pooled estimate?</li></ul><h3>Sources and further reading</h3><ul><li>Aleksander Fabijan, Jayant Gupchup, Somit Gupta, Jeff Omhover, Wen Qin, Lukas Vermeer &amp; Pavel Dmitriev, “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners,” <em>KDD</em>, 2019. <a href="https://doi.org/10.1145/3292500.3330722">https://doi.org/10.1145/3292500.3330722</a></li><li>Microsoft Research, “Diagnosing Sample Ratio Mismatch in A/B Testing.” <a href="https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/">https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/</a></li><li>Ron Kohavi, Diane Tang &amp; Ya Xu, <em>Trustworthy Online Controlled Experiments</em>, Cambridge University Press, 2020, chapters 19, 21 and 23.</li><li>Ron Kohavi, Alex Deng, Roger Longbotham &amp; Ya Xu, “Seven Rules of Thumb for Web Site Experimenters,” <em>KDD</em>, 2014. <a href="https://doi.org/10.1145/2623330.2623341">https://doi.org/10.1145/2623330.2623341</a></li><li>William Shadish, Thomas Cook &amp; Donald Campbell, <em>Experimental and Quasi-Experimental Designs for Generalized Causal Inference</em>, Houghton Mifflin, 2002.</li><li>Course lecture: <a href="https://youtu.be/UabQhB9OBjU">Validity</a></li><li>Course lecture: <a href="https://youtu.be/abi7CbPD698">Internal Validity</a></li><li>Course lecture: <a href="https://youtu.be/0Z6lSNmLff4">Evaluation Checks</a></li><li>Course lecture: <a href="https://youtu.be/mWk8IIA3IVE">Reliability</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04e9e2925073" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: How Much Evidence Is Enough?]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-how-much-evidence-is-enough-8f4621c685d6?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f4621c685d6</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Fri, 23 Oct 2026 05:01:03 GMT</pubDate>
            <atom:updated>2026-10-23T05:01:03.025Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Power, sample size and duration only make sense after you decide what evidence would actually change the decision</em></h4><p><em>Part 8 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0UcgofeUISWtOmXQJ3q14Q.png" /></figure><p>Open almost any A/B-test calculator and it asks for familiar inputs:</p><p>baseline conversion,</p><p>minimum detectable effect,</p><p>alpha,</p><p>power.</p><p>Enter four numbers.</p><p>Get a sample size.</p><p>It feels wonderfully objective.</p><p>Except one of those inputs is hiding a management decision inside a statistical box.</p><p><strong>Minimum detectable effect: 0.5 percentage points.</strong></p><p>Why 0.5?</p><p>Would the company ship at +0.4?</p><p>Would finance care at +0.2?</p><p>Would +0.05 pay for the engineering work?</p><p>The calculator cannot know.</p><p>Yet that choice can multiply or divide the traffic requirement several times over.</p><p>This is the uncomfortable thing about sample-size planning:</p><p>the mathematics may begin in the calculator, but the experiment has already been designed before you open it.</p><p>[ADD VISUALIZATION: a sample-size calculator screen with four inputs: baseline conversion, MDE, alpha, power. Highlight MDE and pull it outward into a business decision bubble: “What is the smallest lift that would actually change what we do?” Below, show two choices for the same checkout experiment: MDE = 0.5 percentage points → feasible sample; MDE = 0.1 percentage points → dramatically larger sample. Caption: “the calculator computes the consequence; it does not choose what matters”.]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVm8_9nLgosQp25KCRmeDw.png" /></figure><h3>Statistical significance is not a decision rule</h3><p>Consider two experiments that both report:</p><p><strong>+2%, p = 0.06.</strong></p><p>One changes the colour of a button on a low-traffic settings page.</p><p>The other changes tax presentation at checkout.</p><p>Treating both as automatic holds because neither crossed 0.05 assumes that the threshold should decide the business action.</p><p>Now add context.</p><p>The button change creates essentially no business value at +2%.</p><p>The PM would not prioritize it even at p = 0.001.</p><p>The checkout change could represent material revenue and is reversible enough to launch cautiously with monitoring.</p><p>Same statistical output.</p><p>Different decisions.</p><p>The point is not that 0.10 is secretly the correct threshold for revenue experiments.</p><p>It is that statistical error tolerances belong in the design, where they can be justified against the consequences of being wrong.</p><p>They do not belong in a post-result negotiation.</p><p>If you see p = 0.06 and only then decide that this experiment deserves alpha = 0.10, you did not make the test risk-sensitive.</p><p>You made the threshold result-sensitive.</p><h3>The p-value is narrower than the job people give it</h3><p>A p-value tells you how surprising data at least this extreme would be under the null hypothesis.</p><p>It does not tell you:</p><p>the probability that the null is true,</p><p>the probability that treatment works,</p><p>the probability the result happened “by chance,”</p><p>or whether the effect matters.</p><p>This produces two common mistakes in opposite directions.</p><p>A very large experiment can make a commercially irrelevant effect statistically convincing.</p><p>A small experiment can produce a non-significant estimate for an effect that would have mattered enormously.</p><p>So the sentence:</p><p><strong>“The experiment was not significant.”</strong></p><p>does not complete the thought.</p><p>The next question is:</p><p><strong>What effect sizes was this experiment capable of detecting?</strong></p><h3>Start sample size from the decision</h3><p>The <strong>minimum detectable effect</strong>, or MDE, is the smallest true effect the design is powered to detect at the chosen alpha and power.</p><p>The statistics define what happens once you choose it.</p><p>They do not tell you what value to choose.</p><p>That has to come from the decision.</p><p>Suppose checkout conversion is 10%.</p><p>Finance says a move to 10.5% would be large enough to justify rollout.</p><p>Now +0.5 percentage points is not an arbitrary calculator default.</p><p>It is connected to an action.</p><p>Design the experiment around an effect ten times smaller, and you may spend weeks buying precision nobody intends to use.</p><p>Design it around an effect much larger than the smallest worthwhile gain, and you risk calling a valuable treatment “no effect” simply because the study could not see it.</p><p>The scaling matters too.</p><p>Under the usual approximations, required sample size is inversely proportional to the square of the effect size.</p><p>Double the MDE, and the required sample can fall to roughly one quarter, all else equal.</p><p>That is why tiny changes in the MDE box can produce enormous changes in the calendar.</p><h3>There are two ways to be wrong</h3><p>Experiment design prices two errors.</p><p><strong>Type I:</strong> you act as though an effect exists when it does not.</p><p><strong>Type II:</strong> you fail to detect an effect that does exist.</p><p>Alpha constrains the first.</p><p>Beta describes the second.</p><p>Power is 1 - beta.</p><p>Those definitions are simple.</p><p>Choosing the values is not.</p><p>A false launch on a cheap, reversible UI treatment may be recoverable.</p><p>A false launch in billing, safety or accessibility can have a very different cost.</p><p>A false hold can also be expensive if the treatment would have generated meaningful value.</p><p>The important discipline is pre-specification.</p><p>If error costs justify different settings for different decisions, write that reasoning before the result exists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56B1WXHK5T6jWHLmpt9IGA.png" /></figure><h3>More traffic is not the only route to more power</h3><p>Teams often translate “underpowered” into “wait longer.”</p><p>Sometimes that is exactly right.</p><p>But precision has other levers.</p><p>You can increase the number of independent units.</p><p>You can keep allocation balanced.</p><p>You can use a less variable outcome.</p><p>And you can remove predictable variation using information collected before treatment.</p><p>That last idea is what makes <strong>CUPED</strong> useful.</p><p>Suppose users differ dramatically in spending during the experiment.</p><p>Some of that difference has nothing to do with treatment.</p><p>High-spending users were already high-spending users last week.</p><p>If pre-period spending predicts experimental-period spending, CUPED can use it as a covariate and remove part of that predictable variance.</p><p>The original Microsoft paper reported variance reductions around 45–52% for important Bing metrics using one week of pre-experiment data.</p><p>That does not mean standard errors fell by 45–52%; standard error scales with the square root of variance.</p><p>But sample-size requirements scale with variance, so variance reduction of that magnitude can translate into a very substantial reduction in required traffic under comparable assumptions.</p><p>There is no free precision.</p><p>If the pre-period variable barely predicts the outcome, CUPED barely helps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1MfkwOGZLAZdU140r8EUeg.png" /></figure><h3>Duration is not whatever remains after sample size</h3><p>A test can have enough observations and still have run for too little time.</p><p>Weekly cycles matter.</p><p>A Tuesday-to-Thursday experiment may have learned about Tuesday-to-Thursday users.</p><p>Novelty matters.</p><p>A redesigned experience may trigger curiosity during the first few days that disappears once people adapt.</p><p>But longer is not automatically better either.</p><p>Every extra day has opportunity cost.</p><p>Some users may continue receiving an inferior version.</p><p>The next experiment waits for traffic.</p><p>A business decision waits for evidence.</p><p>The goal is not to run as long as possible.</p><p>It is to run long enough for the estimate to correspond to the behaviour you actually care about.</p><h3>Your team is going to peek</h3><p>Here is a more realistic experiment protocol:</p><p>Monday: p = 0.21.</p><p>Tuesday: 0.12.</p><p>Wednesday: 0.07.</p><p>Thursday morning:</p><p><strong>0.049.</strong></p><p>Somebody posts the screenshot in Slack.</p><p>Do you stop?</p><p>If the test was designed as a conventional fixed-horizon test, repeatedly checking and stopping the first time p falls below 0.05 changes its error properties.</p><p>Every look creates another opportunity to cross the threshold by chance.</p><p>This is why “just don’t look” is such an unsatisfying organizational solution.</p><p>The dashboard exists precisely because people want to look.</p><p>Sequential methods attack the actual problem.</p><p>Always-valid p-values and related procedures are designed to remain valid under repeated monitoring when their assumptions and stopping rules are respected.</p><p>They do not make arbitrary peeking safe.</p><p>They change the statistical procedure so repeated monitoring is part of the plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wWOTbuVFrdWn_ISQdCRCZA.png" /></figure><p>The organizational lesson is useful:</p><p>if repeated monitoring is inevitable, design for repeated monitoring.</p><p>Do not build a procedure that requires people to behave in a way you already know they will not.</p><h3>One more fact can change the stakes</h3><p>Return to the supposedly harmless button-color experiment.</p><p>Now learn that the new color fails the contrast requirement your accessibility review applies.</p><p>The change is no longer merely a low-value UI tweak.</p><p>A different constraint has entered the decision.</p><p>WCAG AA uses a 4.5:1 minimum contrast ratio for normal text. In Europe, EN 301 549 is the central ICT accessibility standard. A new EN 301 549 v4.1.1 using WCAG 2.2 was published in September 2026, but it has not yet replaced the previously harmonized version as the formal legal reference.</p><p>The statistical result did not change.</p><p>The cost of making the wrong decision did.</p><p>That is the broader point.</p><p>Evidence is never “enough” in the abstract.</p><p>It is enough relative to an action, an effect worth detecting, and the consequences of getting that action wrong.</p><h3>What certainty depends on</h3><p><strong>Business threshold.</strong> What is the smallest effect worth acting on?</p><p><strong>MDE.</strong> What effect is the experiment actually designed to detect?</p><p><strong>Error costs.</strong> What does a false launch cost? What does a missed win cost?</p><p><strong>Power.</strong> How likely are you to detect the target effect if it exists?</p><p><strong>Variance.</strong> How noisy is the outcome?</p><p><strong>Pre-period information.</strong> Can you remove predictable variation?</p><p><strong>Duration.</strong> What calendar behaviour must the experiment experience?</p><p><strong>Monitoring.</strong> Is the procedure fixed-horizon or designed for repeated looks?</p><p>Bandits solve a related but different problem.</p><p>They are attractive when you want to allocate more traffic to better-performing options while learning.</p><p>That does not automatically make them the right tool when the final requirement is a defensible causal effect estimate.</p><h3>Try it in code</h3><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/unit-08-power-mde-demo.ipynb">The first notebook</a> starts with a finance decision rule: ship only if conversion improves by 0.5 percentage points. It works backward from that business threshold to sample size and a power curve, including the quadratic relationship between MDE and required traffic.</p><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/v2-cuped-demo.ipynb">The second notebook</a> shows CUPED buying substantial power when pre-period behavior predicts the outcome, then losing that advantage when the correlation becomes weak.</p><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/v2-peeking-sequential-demo.ipynb">The third notebook</a> simulates 2,000 A/A experiments with no true treatment effect. Repeatedly checking an ordinary test and stopping at the first p-value below 0.05 produces far too many apparent discoveries; the notebook then compares this with a sequential procedure designed for monitoring.</p><p><a href="https://github.com/dinbav/digital-experiments-course/tree/main/notebooks">Blank versions</a> contain the same setups with the analysis left open.</p><h3>Questions to think about</h3><ul><li>What is the smallest effect that would actually change a product decision in your organization?</li><li>Was that number chosen before or after someone opened the sample-size calculator?</li><li>Which is more expensive for this decision: a false launch or a missed win?</li><li>Someone will look at the dashboard tomorrow. Is your inference procedure designed for that?</li></ul><h3>Sources and further reading</h3><ul><li>Alex Deng, Ya Xu, Ron Kohavi &amp; Toby Walker, “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data,” <em>WSDM</em>, 2013. <a href="https://doi.org/10.1145/2433396.2433413">https://doi.org/10.1145/2433396.2433413</a></li><li>Ramesh Johari, Pete Koomen, Leonid Pekelis &amp; David Walsh, “Peeking at A/B Tests: Why It Matters, and What to Do About It,” <em>KDD</em>, 2017. <a href="https://doi.org/10.1145/3097983.3097992">https://doi.org/10.1145/3097983.3097992</a></li><li>Ron Kohavi, Alex Deng, Roger Longbotham &amp; Ya Xu, “Seven Rules of Thumb for Web Site Experimenters,” <em>KDD</em>, 2014. <a href="https://doi.org/10.1145/2623330.2623341">https://doi.org/10.1145/2623330.2623341</a></li><li>Ron Kohavi, Diane Tang &amp; Ya Xu, <em>Trustworthy Online Controlled Experiments</em>, Cambridge University Press, 2020, chapters 15 and 17.</li><li>Ronald Wasserstein &amp; Nicole Lazar, “The ASA Statement on p-Values: Context, Process, and Purpose,” <em>The American Statistician</em> 70(2), 2016. <a href="https://doi.org/10.1080/00031305.2016.1154108">https://doi.org/10.1080/00031305.2016.1154108</a></li><li>DoorDash Engineering, “Improving Experimental Power through Control Using Predictions as Covariate (CUPAC),” 2020. <a href="https://careersatdoordash.com/blog/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/">https://careersatdoordash.com/blog/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/</a></li><li>European Commission / AccessibleEU, “The European accessibility standard EN 301 549 has been updated,” 7 September 2026. <a href="https://accessible-eu-centre.ec.europa.eu/content-corner/news/european-accessibility-standard-en-301-549-has-been-updated-2026-09-07_en">https://accessible-eu-centre.ec.europa.eu/content-corner/news/european-accessibility-standard-en-301-549-has-been-updated-2026-09-07_en</a></li><li>W3C, <em>WCAG 2 Overview</em>. <a href="https://www.w3.org/WAI/standards-guidelines/wcag/">https://www.w3.org/WAI/standards-guidelines/wcag/</a></li><li>Course lecture: <a href="https://youtu.be/v3th5hLM3p8">Hypothesis Testing and Evaluation</a></li><li>Course lecture: <a href="https://youtu.be/Qk_j8KtAsUM">Business Interpretation</a></li><li>Course lecture: <a href="https://youtu.be/BMmbADxBffA">Regression Analysis and P-Hacking</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f4621c685d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: You’re Not Testing What You Think]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-youre-not-testing-what-you-think-7f8c066bcf3e?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f8c066bcf3e</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[personalization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 05:01:02 GMT</pubDate>
            <atom:updated>2026-10-20T05:01:02.211Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Treatment design is the contract between what your hypothesis says, what production delivers, and what your analysis eventually claims</em></h4><p><em>Part 7 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MkhfWncyRGslJM5r1CIuFg.png" /></figure><p>The experiment says:</p><p><strong>personalized recommendations vs. control.</strong></p><p>Clean enough.</p><p>Now ask the engineer what “personalized” means.</p><p>One system says:</p><blockquote><em>If the customer bought running shoes, show running socks.</em></blockquote><p>Another calls a model on every request, passing current inventory, browsing history and a catalog that changes throughout the day.</p><p>Both teams can write <strong>personalized recommendations</strong> in the experiment dashboard.</p><p>They are not testing the same intervention.</p><p>That distinction sounds semantic until the experiment ends and someone asks the only question that matters:</p><p><strong>What exactly did treatment users receive?</strong></p><p>If the answer starts with “it depends,” treatment design was part of the experiment whether you planned for it or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RTye1vxZRasEGOdtb5nmyg.png" /></figure><h3>The Jira ticket is not the treatment</h3><p>Experiment documents are full of adjectives:</p><p>better onboarding,</p><p>smarter ranking,</p><p>more relevant advertising,</p><p>simpler checkout,</p><p>personalized recommendations.</p><p>They describe an intention.</p><p>A treatment specification describes delivery.</p><p>For a deterministic recommendation rule, that may be easy:</p><blockquote><em>Users who purchased running shoes see running socks in slot two of the home-page recommendation module.</em></blockquote><p>Now change the system to a model.</p><p>Suddenly the specification needs more:</p><p>Which model version?</p><p>Which prompt or feature set?</p><p>Which temperature?</p><p>Which catalog?</p><p>Which inventory state?</p><p>Which fallback?</p><p>What happens for a user with no history?</p><p>What happens when the recommendation service times out?</p><p>The treatment has become a system, not a screen.</p><h3>The forgotten half of SUTVA</h3><p>SUTVA is usually introduced through interference: one unit’s treatment should not change another unit’s outcome.</p><p>Treatment design exposes the other half.</p><p>The intervention itself needs to be sufficiently well defined.</p><p>A blue button is easy to recognize as the same intervention across users.</p><p>A live recommendation model is different.</p><p>Even with random assignment working perfectly, treatment users may receive meaningfully different realizations.</p><p>That does not automatically make the experiment invalid.</p><p>Many useful treatments are heterogeneous by design.</p><p>But it changes what the estimand means.</p><p>You may be estimating the effect of a <strong>policy that generates a distribution of experiences</strong>, rather than the effect of one fixed experience.</p><p>That distinction should be explicit.</p><h3>Freezing the model does not freeze the product</h3><p>Suppose the team pins the model version and sets temperature to zero.</p><p>Problem solved?</p><p>The recommendation still depends on the world around it.</p><p>An item goes out of stock.</p><p>A new product enters the catalog.</p><p>A user has no history.</p><p>The personalization service fails and the page falls back to popularity ranking.</p><p>The code may be stable while delivered treatment continues to vary.</p><p>This is why a useful treatment description includes both the intended path and the fallback path.</p><p>The fallback is not an implementation footnote.</p><p>If enough users see it, it is part of the experimental condition.</p><h3>Personalization changes meaning across user states</h3><p>There is another reason “personalized” hides too much.</p><p>The same delivered intervention can have different value depending on when the user receives it.</p><p>Lambrecht and Tucker studied an online travel firm comparing generic brand advertising with dynamic retargeting: ads showing people products they had already viewed.</p><p>The intuitive prediction is straightforward.</p><p>Specific should beat generic.</p><p>On average, it did not.</p><p>Dynamic retargeting underperformed the generic advertising.</p><p>But that relationship changed when browsing behaviour suggested that users had already narrowed their preferences.</p><p>Once someone was further along in the decision process, specificity stopped carrying the same disadvantage.</p><p>The lesson is not “retargeting works only after review sites.”</p><p>It is that treatment design may need to include <strong>state</strong>.</p><p>Who receives a treatment, and when, can be part of what the treatment is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MNlfBsps0F7WQUx4fiqdcQ.png" /></figure><p>That is very different from discovering the interaction after the experiment and searching through segments until one looks interesting.</p><p>A pre-specified state-dependent policy is treatment design.</p><p>Post-hoc fishing is analysis.</p><h3>A feature flag solves a different problem</h3><p>Production teams need feature flags.</p><p>Flags separate deploying code from releasing it.</p><p>They let a team ship disabled code, ramp exposure gradually, and turn it off quickly.</p><p>Those are valuable capabilities.</p><p>But the interface can create a dangerous illusion.</p><p>A tool shows:</p><p>Control: 95%</p><p>Treatment: 5%</p><p>It looks like an experiment.</p><p>It may only be a rollout.</p><p>If the five percent were selected because they use the newest app version, assignment is confounded with app version.</p><p>If treatment is limited to one country, geography differs by construction.</p><p>If everybody eventually receives the feature and the team compares this week with last week, there is no concurrent control at all.</p><p>The feature flag delivered the code exactly as requested.</p><p>It still did not create causal identification.</p><p>[ADD VISUALIZATION: a horizontal production-to-evidence pipeline for one feature: “Deploy code” → “Feature flag” → “Random assignment” → “Actual exposure” → “Outcome measurement”. Put a bracket only under “Deploy code → Feature flag” labelled “delivery infrastructure”. Put another bracket under “Random assignment → Actual exposure → Outcome measurement” labelled “experimental design and execution”. Add one failed example below: “5% selected by app version” with a red X and caption “ramp ≠ randomization”.]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PGvad6bh_KYRPMHUM2P4tw.png" /></figure><h3>Assignment and exposure are not the same event</h3><p>Even a properly randomized experiment has another implementation question.</p><p>A user can be assigned to treatment and never actually receive it.</p><p>The flag fails to load.</p><p>The client is too old.</p><p>The model times out.</p><p>The recommended item disappears before render.</p><p>An ad blocker removes the surface.</p><p>This creates a practical distinction between:</p><p><strong>assignment</strong></p><p>and</p><p><strong>exposure</strong>.</p><p>The analysis strategy depends on what question you are answering, but the design problem comes first: you need to know what failure modes exist and instrument them.</p><p>Otherwise “treatment” becomes shorthand for a mixture of people who experienced the intervention and people who did not.</p><h3>Access defines the intervention space</h3><p>Before designing the perfect treatment, ask what you are actually allowed to change.</p><p>On your own product, the intervention space may be broad.</p><p>Inside a partner integration, perhaps only one component is under your control.</p><p>Through a public API, the API defines the available actions.</p><p>Without cooperation from the platform, you may be acting only as an ordinary user.</p><p>This is not a ranking from rigorous to weak.</p><p>It is a constraint map.</p><p>Different access gives you different possible treatments.</p><h3>“User” is also an implementation choice</h3><p>Identification complicates treatment delivery further.</p><p>Logged-in products have a relatively stable account identifier.</p><p>Open web traffic is messier.</p><p>Safari and Firefox restrict third-party cookies by default.</p><p>In April 2025, Google decided not to proceed with the planned standalone third-party-cookie prompt in Chrome and maintained Chrome’s existing user-choice approach.</p><p>By 2026, Privacy Sandbox itself is no longer a single replacement path moving in one direction: some technologies such as CHIPS and FedCM continue to be supported, while several advertising-related APIs are scheduled for deprecation or removal.</p><p>For experimentation, the relevant lesson is not the browser-policy history.</p><p>It is this:</p><p><strong>“user-level treatment” may not mean the same thing across all of your traffic.</strong></p><p>Identity is part of delivery.</p><h3>Some treatments should fail before the experiment starts</h3><p>Treatment design also has a boundary that statistics cannot rescue.</p><p>Suppose the team wants to optimize a cookie banner.</p><p>One variant has a large, bright Accept button and a visually buried rejection option.</p><p>It may increase consent clicks.</p><p>That does not automatically make it an admissible treatment.</p><p>European data-protection guidance emphasizes that consent must be freely given. The EDPB has specifically noted that pre-ticked opt-in boxes do not produce valid consent, and that interfaces which clearly push users toward acceptance can undermine the validity of consent.</p><p>The optimization problem therefore starts after the compliance boundary, not before it.</p><p>Fake-door testing creates a different version of the same discipline.</p><p>A button for a feature that does not exist can measure expressed interest.</p><p>It does not measure the causal effect that the finished feature would have on retention, revenue or satisfaction.</p><p>A good experiment statement says what evidence the treatment can produce, not what the team hopes it will eventually imply.</p><h3>The treatment is a contract</h3><p>By the time results arrive, three groups need to mean the same thing when they say “treatment”:</p><p>the person who wrote the hypothesis,</p><p>the system that delivered the experience,</p><p>and the analyst interpreting the outcome.</p><p>That alignment is the treatment specification.</p><p>It should answer:</p><p>What changes?</p><p>For whom?</p><p>Through which channel?</p><p>Under which state?</p><p>What remains fixed?</p><p>What is allowed to adapt?</p><p>What happens on failure?</p><p>A treatment is not the wireframe.</p><p>It is not the feature name.</p><p>It is not the flag.</p><p>It is the operational definition of what changed in the user’s world.</p><h3>What treatment design depends on</h3><p><strong>Description.</strong> Can another team implement it without guessing?</p><p><strong>Adaptation.</strong> Is it a fixed rule or a policy generating a distribution of experiences?</p><p><strong>State.</strong> Which inventory, catalog, history or context can alter delivery?</p><p><strong>Exposure.</strong> What does the user actually receive after assignment?</p><p><strong>Channel.</strong> Server-side, client-side, email, app, partner integration?</p><p><strong>Access.</strong> Which parts of the system can you manipulate?</p><p><strong>Identity.</strong> What does “same user” mean across your traffic?</p><p><strong>Compliance.</strong> Is the proposed treatment admissible before anyone optimizes it?</p><h3>Questions to think about</h3><ul><li>Rewrite one treatment in your current roadmap without using an adjective such as “better,” “smarter” or “personalized.”</li><li>What can still drift after your code and model version are frozen?</li><li>What experience does a treatment user receive when delivery fails?</li><li>Which rollout in your organization looks like an experiment in the dashboard but does not actually contain random assignment?</li></ul><h3>Sources and further reading</h3><ul><li>Anja Lambrecht &amp; Catherine Tucker, “When Does Retargeting Work? Information Specificity in Online Advertising,” <em>Journal of Marketing Research</em> 50(5), 2013, 561–576. <a href="https://doi.org/10.1509/jmr.11.0503">https://doi.org/10.1509/jmr.11.0503</a></li><li>Robert M. Bond, Christopher J. Fariss, Jason J. Jones, Adam D. I. Kramer, Cameron Marlow, Jaime E. Settle &amp; James H. Fowler, “A 61-Million-Person Experiment in Social Influence and Political Mobilization,” <em>Nature</em> 489, 2012. <a href="https://doi.org/10.1038/nature11421">https://doi.org/10.1038/nature11421</a></li><li>Ron Kohavi, Diane Tang &amp; Ya Xu, <em>Trustworthy Online Controlled Experiments</em>, Cambridge University Press, 2020, chapter 4.</li><li>EDPB, <em>Guidelines 03/2022 on Deceptive Design Patterns in Social Media Platform Interfaces</em>, v2.0, 2023. <a href="https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-032022-deceptive-design-patterns-social-media_en">https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-032022-deceptive-design-patterns-social-media_en</a></li><li>EDPB, <em>Report of the work undertaken by the Cookie Banner Taskforce</em>, 18 January 2023. <a href="https://www.edpb.europa.eu/system/files/2023-01/edpb_20230118_report_cookie_banner_taskforce_en.pdf">https://www.edpb.europa.eu/system/files/2023-01/edpb_20230118_report_cookie_banner_taskforce_en.pdf</a></li><li>Google, “Next steps for Privacy Sandbox and tracking protections in Chrome,” 22 April 2025. <a href="https://privacysandbox.google.com/blog/privacy-sandbox-next-steps">https://privacysandbox.google.com/blog/privacy-sandbox-next-steps</a></li><li>Google, “Privacy Sandbox feature status,” updated 2026. <a href="https://privacysandbox.google.com/overview/status">https://privacysandbox.google.com/overview/status</a></li><li>Course lecture: <a href="https://youtu.be/neIshaqbsrA">Designing the Experiment</a></li><li>Course lecture: <a href="https://youtu.be/j9MEIQ3QKsk">Technologies for Intervention</a></li><li>Course lecture: <a href="https://youtu.be/fKN7kvbhpHE">Additional Design Aspects</a></li><li>Course lecture: <a href="https://youtu.be/h9voMBl78MI">Fake Door Testing</a></li></ul><p><em>A sourcing note:</em> the hotel-form +52% result, the Citi Simplicity case and the Financial Times fake-door figures appear in the course material but do not have sufficiently strong public sources, so they are not used here as numerical evidence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f8c066bcf3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: Your Control Group Isn’t a Control Group]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-your-control-group-isnt-a-control-group-df9e7155e9ac?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/df9e7155e9ac</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 05:01:04 GMT</pubDate>
            <atom:updated>2026-10-16T05:01:04.332Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Assignment mechanisms, interference, and SUTVA — when randomizing by user quietly corrupts what you measured</em></h4><p><em>Part 6 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_hAPSvHS9dSdGQ-Goyy8Zg.png" /></figure><p>Your assignment is random.</p><p>The groups are balanced.</p><p>Instrumentation is clean.</p><p>The confidence interval is narrow.</p><p>Treatment wins.</p><p>And the experiment can still be wrong.</p><p>The missing question is:</p><p><strong>Did treatment change the world experienced by control?</strong></p><p>Imagine a food-delivery marketplace testing a faster driver-assignment algorithm.</p><p>Users are randomized individually.</p><p>Treated orders get drivers sooner.</p><p>Delivery time falls.</p><p>On paper, this is a textbook A/B test.</p><p>Now stop looking at the users and look at the driver pool.</p><p>A driver assigned earlier to a treated order is no longer available to a control order.</p><p>The control customer waits longer.</p><p>Not because the old algorithm became worse.</p><p>Because treatment changed the market underneath control.</p><p>The control arm is randomized.</p><p>It just is not a control condition anymore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VWyr80v2BsBlq9TV30X0_w.png" /></figure><p>That failure has a name:</p><p><strong>interference</strong>.</p><h3>Randomization does not create independence</h3><p>Random assignment protects us from systematic differences between groups at baseline.</p><p>It does not guarantee that the groups remain causally isolated after assignment.</p><p>That distinction matters in systems with shared resources:</p><p>drivers,</p><p>inventory,</p><p>feed slots,</p><p>support agents,</p><p>buyers and sellers,</p><p>ranking signals,</p><p>conversation context.</p><p>If treatment applied to one unit changes the outcome of another unit, the usual no-interference assumption has failed.</p><p>That assumption is part of <strong>SUTVA</strong>, the Stable Unit Treatment Value Assumption.</p><p>In practical terms, SUTVA asks for two things:</p><p>a well-defined, consistent treatment,</p><p>and no cross-unit interference.</p><p>Marketplace experiments often fail on the second.</p><h3>Why the estimate can look unusually convincing</h3><p>Suppose the new matching algorithm genuinely improves treated orders.</p><p>At the same time, treatment consumes scarce supply faster and makes control orders worse.</p><p>The observed gap now grows from both sides.</p><p>Treatment improves.</p><p>Control deteriorates.</p><p>The experiment may therefore estimate something different from the rollout effect the team actually cares about.</p><p>There is no universal correction term hiding in the dashboard.</p><p>Interference can be modeled under explicit assumptions, and some methods are designed to estimate effects in its presence.</p><p>But without a design or model that describes the interference structure, you cannot simply subtract “spillover bias” from the lift.</p><p>And increasing the number of users does not repair the identification problem.</p><p>More observations can reduce sampling uncertainty around an estimate whose causal interpretation is already compromised.</p><p>This is the same distinction that appears throughout experimentation:</p><p><strong>precision is not identification.</strong></p><h3>The same randomization can be valid at lunch and invalid at dinner</h3><p>This is why rules such as “always randomize by user” are dangerous.</p><p>Suppose the marketplace has abundant supply at lunchtime.</p><p>There are enough drivers that assigning one slightly earlier has essentially no effect on everyone else.</p><p>A rider-level split may be fine.</p><p>Run the same experiment during a severe supply shortage.</p><p>Now one treated request can materially change waiting time for other riders.</p><p>Same product.</p><p>Same code.</p><p>Same randomization unit.</p><p>Different interference structure.</p><p>The correct design changed because the system changed.</p><p>The randomization unit is therefore not merely an implementation detail.</p><p>It is part of the causal model.</p><h3>Change what gets randomized</h3><p>Teams operating large marketplaces did not conclude that experimentation was impossible.</p><p>They changed the unit.</p><p>One common solution is a <strong>switchback experiment</strong>.</p><p>Instead of assigning individual users to treatment or control, assign an entire market, region or time block to one condition at a time.</p><p>A region may run treatment during one interval and control during another, alternating repeatedly.</p><p>Now users competing for the same scarce resource are exposed to the same condition at the same moment.</p><p>The treatment arm is no longer draining supply from a simultaneous control arm inside the same market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wBvY9YWsgbBrBGsBhYSKfg.png" /></figure><p>DoorDash and Lyft have both published about time-based or market-level experimental designs motivated by marketplace interference.</p><p>But the solution is not free.</p><h3>Switchbacks trade bias for a harder experiment</h3><p>The first cost is <strong>effective sample size</strong>.</p><p>A million user events do not give you a million independent assignments when treatment changes only across a few dozen market-time blocks.</p><p>The number of independent units is much closer to the number of blocks.</p><p>Standard errors need to respect that structure.</p><p>The second cost is <strong>carryover</strong>.</p><p>A user who experiences control at 18:55 may change behaviour during the treatment block at 19:05.</p><p>Orders persist.</p><p>Drivers reposition.</p><p>Queues clear slowly.</p><p>The previous condition can leak into the next.</p><p>The third cost is <strong>time horizon</strong>.</p><p>Switchbacks are especially useful for short-run marketplace outcomes.</p><p>They are much less natural for outcomes that need a persistent untreated population over weeks or months.</p><p>And the fourth cost is product experience.</p><p>Rapidly toggling a visible user-facing feature can create a treatment nobody would ever actually ship.</p><p>So “interference exists” does not imply “always use a switchback.”</p><p>It means the assignment unit has to match the interference structure.</p><h3>Identity is not a randomization unit</h3><p>Digital systems make it easy to confuse what we can identify with what we should randomize.</p><p>A cookie is an identifier.</p><p>A session is an identifier.</p><p>An account is an identifier.</p><p>None is automatically the correct experimental unit.</p><p>A session-level experiment may maximize event count while exposing the same person to both variants.</p><p>An account-level assignment may preserve consistency but fail when people share devices or households.</p><p>A conversational product may generate thousands of message turns, but turn-level randomization is usually nonsensical when earlier turns remain inside the model context for later ones.</p><p>The relevant question is not:</p><p><strong>What ID does the experimentation platform expose?</strong></p><p>It is:</p><p><strong>At what level can treatment be changed without contaminating another assigned unit?</strong></p><h3>Blocking solves a different problem</h3><p>Not every assignment design that looks more sophisticated is about interference.</p><p>Sometimes the goal is simply better balance and lower variance.</p><p>With <strong>blocking</strong>, units are grouped before randomization according to a known covariate.</p><p>With <strong>matched pairs</strong>, similar units are paired and treatment is randomized within each pair.</p><p>Those are design-time decisions.</p><p>They are fundamentally different from slicing the results into subgroups after the experiment until one segment produces an interesting p-value.</p><p>One imposes structure before observing outcomes.</p><p>The other searches outcomes for structure afterwards.</p><h3>Shared state can be social, not physical</h3><p>A driver pool makes interference easy to see because the shared resource is tangible.</p><p>Digital products also create shared informational environments.</p><p>Muchnik, Aral and Taylor randomly assigned more than 100,000 newly posted comments to receive one artificial positive vote, one artificial negative vote, or no manipulation.</p><p>A single positive nudge increased the probability of a later positive rating by 32 percent, and the advantage accumulated into final ratings roughly 25 percent higher on average.</p><p>That experiment is not the same interference problem as the driver marketplace.</p><p>The social response to the manipulated score was the mechanism being studied on the treated item itself.</p><p>But it demonstrates an important product-design fact:</p><p>visible scores alter subsequent behaviour.</p><p>Ratings,</p><p>rankings,</p><p>“popular now” shelves,</p><p>recommendation feeds,</p><p>and social proof can become part of the environment that later users respond to.</p><p>When the product state itself carries treatment forward, independence becomes an assumption that deserves inspection rather than a default.</p><h3>SUTVA is not a diagnostic test</h3><p>You cannot finish an experiment and run a statistical test for SUTVA.</p><p>It is an assumption about the data-generating process.</p><p>Did one assigned unit change another unit’s treatment or outcome?</p><p>Was the treatment the same thing each time it was delivered?</p><p>Those questions belong before launch.</p><p>If the answer changes the design, that is exactly the point.</p><h3>The useful design question</h3><p>Before choosing a randomization unit, name the shared state.</p><p>Is it:</p><p>a driver pool?</p><p>inventory?</p><p>a queue?</p><p>a ranking?</p><p>a household?</p><p>a model conversation?</p><p>a market price?</p><p>Then ask:</p><p><strong>Who can change that state for whom?</strong></p><p>That answer usually tells you far more about the correct assignment unit than the experimentation platform’s default.</p><h3>See the bias in code</h3><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/unit-06-randomization-demo.ipynb">This notebook</a> simulates a ride-hailing city where riders assigned to treatment and control share the same driver pool.</p><p>A user-level A/B test reports a lift about three times the true effect of a full rollout, while control riders become worse off than they would have been without the experiment.</p><p>A switchback design, where entire days are assigned to treatment or control, lands much closer to the rollout effect.</p><p>It also shows the price of respecting the real assignment unit:</p><p>the honest standard error is much wider than the one obtained by pretending every rider-day is independent.</p><p>For readers who want to work through the analysis themselves, <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/notebooks/unit-06-randomization-exercise.ipynb">the blank version</a> contains the same setup with the analysis left open.</p><h3>Questions to think about</h3><ul><li>Who else receives part of your treatment besides the unit you explicitly assigned?</li><li>What shared resource or shared state connects your treatment and control arms?</li><li>Would the same randomization unit still be valid under peak load?</li><li>If you move to clusters or switchbacks, what question becomes harder to answer?</li></ul><h3>Sources and further reading</h3><ul><li>Muchnik, Aral &amp; Taylor, “Social Influence Bias: A Randomized Experiment,” <em>Science</em> 341(6146), 2013, 647–651. <a href="https://doi.org/10.1126/science.1240466">https://doi.org/10.1126/science.1240466</a></li><li>DoorDash Engineering, “Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash,” 2018. <a href="https://doordash.engineering/2018/02/13/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash/">https://doordash.engineering/2018/02/13/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash/</a></li><li>DoorDash Engineering, “Analyzing Switchback Experiments by Cluster Robust Standard Error,” 2019. <a href="https://doordash.engineering/2019/09/11/cluster-robust-standard-error-in-switchback-experiments/">https://doordash.engineering/2019/09/11/cluster-robust-standard-error-in-switchback-experiments/</a></li><li>Lyft Engineering, “Experimentation in a Ridesharing Marketplace.” <a href="https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-b39db027a66e">https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-b39db027a66e</a></li><li>Bright, Delarue &amp; Lobel, “Reducing Marketplace Interference Bias Via Shadow Prices,” <em>EC</em>, 2023. <a href="https://arxiv.org/abs/2205.02274">https://arxiv.org/abs/2205.02274</a></li><li>Kohavi, Tang &amp; Xu, <em>Trustworthy Online Controlled Experiments</em>, Cambridge University Press, 2020, chapters 14 and 18.</li><li>Course lecture: <a href="https://youtu.be/xln7hxmJkvo">Assignment Mechanism</a>.</li><li>Course lecture: <a href="https://youtu.be/d9zm6-WQrps">Advanced Assignment Designs</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df9e7155e9ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: Not Every Experiment Is an A/B Test]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-not-every-experiment-is-an-a-b-test-458159b78e15?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/458159b78e15</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 05:01:02 GMT</pubDate>
            <atom:updated>2026-10-13T05:01:02.268Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>What to do when the intervention already shipped, randomization is impossible, or the system itself refuses to stay inside treatment and control</em></h4><p><em>Part 5 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QCKZZk5HDrcJi9t_z_Ny0Q.png" /></figure><p>The pricing page went live last Tuesday.</p><p>Everyone has it.</p><p>Leadership now wants to know whether it increased revenue.</p><p>Preferably causally.</p><p>This is the point where someone usually says:</p><p>“Can we A/B test it?”</p><p>No.</p><p>Not anymore.</p><p>You cannot randomize yesterday.</p><p>And that simple constraint exposes something important about experimentation:</p><p><strong>A/B testing is one experimental design tool. It is not the definition of causal evidence.</strong></p><p>Sometimes randomization is available.</p><p>Sometimes it is available, but only at the wrong unit.</p><p>Sometimes the world assigns treatment for you.</p><p>Sometimes the intervention has already happened.</p><p>And sometimes the honest answer is that no available design supports the causal claim you want to make.</p><p>The difficult part of experimentation is often not running the test.</p><p>It is discovering what kind of comparison the system still allows.</p><h3>Start with what you control</h3><p>Take the same business question:</p><p><strong>Did the new pricing page increase revenue?</strong></p><p>Now place it in four different systems.</p><p>On your own website, before launch, you can randomize users or accounts.</p><p>On a partner’s website, perhaps you can randomize what your injected script displays, while caching and incomplete exposure remain outside your control.</p><p>Through a public API, your treatment and measurement options are whatever the interface exposes.</p><p>After the page has shipped universally, individual randomization is gone.</p><p>The question has not changed.</p><p>The design has.</p><p>This is why “Which experiment should we run?” is downstream of another question:</p><p><strong>What variation can we actually create or credibly exploit?</strong></p><h3>Sometimes there is no rescue design</h3><p>Suppose the rollout happened everywhere on the same day.</p><p>There is no untreated market.</p><p>No staggered rollout.</p><p>No relevant threshold.</p><p>No comparable region.</p><p>And at the same time, the company changed prices, launched a campaign, and entered a seasonal peak.</p><p>You can still draw a before-and-after chart.</p><p>You can still calculate a difference.</p><p>What you may not have is a credible counterfactual.</p><p>That matters because a causal estimate is not produced by attaching a statistical method to a time series.</p><p>It comes from a design that makes the missing outcome believable enough to estimate.</p><p>Sometimes the most professional answer is:</p><blockquote><em>With the variation available here, I cannot identify the causal effect of this rollout.</em></blockquote><p>That sentence protects more credibility than an impressive-looking number generated from a design nobody believes.</p><h3>Randomization is part of a design, not the starting point</h3><p>A useful planning sequence is:</p><p><strong>design type → assignment mechanism → treatment → evaluation</strong></p><p>Starting with “let’s do an A/B test” reverses the logic.</p><p>Consider a marketplace.</p><p>A treatment changes driver behaviour.</p><p>Drivers affect riders.</p><p>Riders affect drivers.</p><p>Randomizing individual users may technically produce two groups while failing to produce independent treatment and control conditions.</p><p>Or consider a purchasing decision that develops over three weeks.</p><p>Randomizing sessions may be easy.</p><p>It may also be the wrong unit entirely.</p><p>The assignment mechanism has to match the system that generates the outcome.</p><h3>When the world assigns treatment</h3><p>Not every useful treatment assignment comes from an experimenter.</p><p>A policy begins in one jurisdiction.</p><p>A regulation applies above an eligibility cutoff.</p><p>An outage affects one region.</p><p>A lottery allocates access.</p><p>These situations may create a <strong>natural experiment</strong>.</p><p>The attraction is obvious.</p><p>The variation already exists.</p><p>You do not need to expose anyone deliberately.</p><p>Sometimes it is the only route to the question.</p><p>But “natural” does not mean “random.”</p><p>A company policy applied to some departments may also select departments with different customers, budgets, managers, or growth trajectories.</p><p>A geographic boundary may separate populations that differ for many reasons besides treatment.</p><p>The causal argument rests on why the external assignment mechanism creates a comparison that is credible enough for the estimand you care about.</p><p>A lottery can make that argument unusually strong.</p><p>A sharp administrative cutoff may make local comparisons plausible.</p><p>An arbitrary boundary may create something close to as-if random exposure.</p><p>But the assignment story is doing the causal work.</p><p>More rows do not strengthen a weak story into a randomized one.</p><h3>Quasi-experiments make assumptions visible</h3><p>A large amount of real experimentation work lives between a textbook RCT and a naive before/after comparison.</p><p>That territory is usually called <strong>quasi-experimental design</strong>.</p><p>Its defining feature is not that it is “less scientific.”</p><p>It is that identification relies on structural assumptions other than researcher-controlled individual randomization.</p><p>And those assumptions have names.</p><h3>Difference-in-differences</h3><p>Suppose a policy changes in treated cities but not in comparison cities.</p><p>Difference-in-differences asks:</p><p>How much did the treated group change relative to how much the comparison group changed?</p><p>The central identifying assumption is <strong>parallel trends</strong>.</p><p>Without treatment, the two groups would have continued to move similarly.</p><p>Pre-treatment trends can make this assumption more or less plausible.</p><p>They cannot show us the untreated post-treatment path of the treated group.</p><p>That path is still counterfactual.</p><h3>Synthetic control</h3><p>Sometimes one treated city has no obvious single control city.</p><p>Synthetic control constructs one.</p><p>It chooses weights over untreated donor units so that their weighted history resembles the treated unit before intervention.</p><p>That synthetic unit then serves as the estimate of what would have happened afterwards without treatment.</p><p>Its credibility depends on good pre-treatment fit, a useful donor pool, and the absence of meaningful contamination or spillover into those donors.</p><h3>Regression discontinuity</h3><p>Suppose treatment turns on when a score crosses 80.</p><p>People at 79.9 and 80.1 may be much more comparable than people at 30 and 90.</p><p>Regression discontinuity exploits that local comparison.</p><p>The design relies on potential outcomes varying smoothly around the cutoff and on units not precisely manipulating which side of the threshold they land on.</p><p>The causal claim is also local:</p><p>it concerns units near the cutoff.</p><h3>Interrupted time series</h3><p>Sometimes all you have is one outcome observed repeatedly before and after an intervention.</p><p>Interrupted time series models the pre-intervention trajectory and asks whether level or slope changes after the intervention.</p><p>It can be useful.</p><p>It is also vulnerable to history.</p><p>If another event occurred at the same time, that event can produce the same apparent interruption.</p><p>A suitable comparison series can strengthen the design substantially.</p><h3>Production systems force these choices</h3><p>These are not exotic methods waiting in econometrics textbooks.</p><p>Companies use them because production systems create exactly these constraints.</p><p>Netflix described its quasi-experimentation work in 2018 for cases where individual-level A/B testing was not feasible.</p><p>TV and billboard advertising are obvious examples.</p><p>You cannot expose one member standing on a street corner to a billboard while withholding the same billboard from the person beside them.</p><p>The experimental unit therefore shifts toward geography.</p><p>That creates a new problem.</p><p>A consumer product may have millions of users but only a small number of usable geographic units.</p><p>Netflix’s 2020 discussion of quasi-experiment challenges explicitly highlights small geographic sample sizes, limited historical observations, and differences between markets.</p><p>Scale at the user level does not rescue you when the assignment unit is the city.</p><p>Uber ran into a different version of the same structural problem.</p><p>Riders and drivers interact.</p><p>Treating part of a marketplace can change conditions for the nominal control group through network effects.</p><p>Uber engineers described situations where an entire market has to receive treatment, and the comparison is instead constructed from untreated donor cities using synthetic control.</p><p>A/B testing did not “fail” in either case.</p><p>The systems changed what a valid treatment unit could be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DI0tfwQJeafVMVNOqZKftw.png" /></figure><h3>Lab and field experiments buy different things</h3><p>Randomized experiments themselves are not all the same.</p><p>A laboratory setting buys control.</p><p>You can standardize exposure, measure intermediate variables carefully, and isolate a mechanism.</p><p>But controlled environments can introduce questions about external validity.</p><p>The famous <strong>WEIRD</strong> critique from Henrich, Heine, and Norenzayan highlighted how behavioural research had often generalized broadly from participants drawn disproportionately from Western, Educated, Industrialized, Rich, and Democratic societies. Their argument was not that laboratory experiments are invalid; it was that narrow populations can make broad generalization much harder to defend.</p><p>Field experiments trade some environmental control for behaviour inside the real system.</p><p>That can make them highly valuable for deployment decisions.</p><p>It can also create ethical questions, particularly when people are unaware that their behaviour is part of research.</p><p>The relevant question is not simply whether a team considers the treatment harmless.</p><p>Risk, consent, participant expectations, institutional governance, and the applicable ethical or regulatory framework all matter.</p><p>Good experiment design includes those constraints too.</p><h3>Now return to the pricing page</h3><p>It already shipped.</p><p>So what can you still use?</p><p>Perhaps the rollout happened at different times across countries.</p><p>Perhaps one similar market did not receive the change.</p><p>Perhaps a threshold determined eligibility.</p><p>Perhaps you have a long and stable pre-period that supports a time-series design.</p><p>Perhaps a weighted combination of untreated markets can approximate the treated market.</p><p>Or perhaps none of those is credible.</p><p>The method should come after that inventory.</p><p>Not before it.</p><p>A useful habit is to write the method and its identifying assumption next to each other:</p><p><strong>Difference-in-differences:</strong> parallel trends.</p><p><strong>Synthetic control:</strong> a credible untreated counterfactual constructed from the donor pool.</p><p><strong>Regression discontinuity:</strong> continuity around the threshold and no precise manipulation of assignment.</p><p><strong>Interrupted time series:</strong> the pre-intervention process provides a defensible counterfactual trajectory, without a competing event explaining the break.</p><p>The method name tells you how the estimate is computed.</p><p>The assumption tells you why anyone should interpret it causally.</p><h3>What design choice depends on</h3><p>It depends on:</p><p><strong>Control.</strong> Can you determine who receives treatment?</p><p><strong>Timing.</strong> Has the intervention already happened?</p><p><strong>Unit.</strong> User, account, household, market, city, or time block?</p><p><strong>Interference.</strong> Can treatment in one unit change outcomes in another?</p><p><strong>Comparison.</strong> Is there a credible untreated group, geography, threshold, or time series?</p><p><strong>Measurement.</strong> Do you observe the outcome at the level where treatment operates?</p><p><strong>Ethics and governance.</strong> Is the exposure acceptable, and under what conditions?</p><p>Constraints do not merely make an experiment harder.</p><p>They change the causal sentence the evidence can support.</p><h3>See parallel trends fail in code</h3><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/v2-did-demo.ipynb">This notebook</a> simulates a policy introduced in some cities but not others, with a true treatment effect of 5.</p><p>A naive post-treatment comparison returns roughly 12 because the treated cities already had higher outcomes before treatment.</p><p>A treated-group before/after comparison returns roughly 9.5 because it absorbs the common time trend.</p><p>Difference-in-differences recovers roughly 5, with an interval containing the true effect.</p><p>Then the data-generating process changes.</p><p>The treated cities receive their own underlying growth trend, violating parallel trends.</p><p>The same estimator now settles confidently around 7.6.</p><p>That is the failure mode worth experiencing:</p><p><strong>precise and wrong.</strong></p><p>The notebook’s placebo check makes the violation visible before the result is trusted.</p><p>For readers who want to work through the analysis themselves, <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/notebooks/v2-did-exercise.ipynb">the blank version</a> contains the same setup with the analysis left open.</p><h3>Questions to think about</h3><ul><li>For a decision you cannot randomize, what is the strongest design the system still permits?</li><li>Which identifying assumption carries most of your causal claim?</li><li>Where could interference make your control group not truly untreated?</li><li>Which question in your organization should receive the answer: “We currently do not have a credible design”?</li></ul><h3>Sources and further reading</h3><ul><li>Netflix Technology Blog, <a href="https://netflixtechblog.com/quasi-experimentation-at-netflix-566b57d2e362">“Quasi Experimentation at Netflix”</a>, 2018. Why Netflix uses quasi-experiments when individual-level randomization is infeasible.</li><li>Netflix Technology Blog, <a href="https://netflixtechblog.com/key-challenges-with-quasi-experiments-at-netflix-89b4f234b852">“Key Challenges with Quasi Experiments at Netflix”</a>, 2020. Small geographic samples, limited pre-intervention data, and heterogeneity across units.</li><li>Nick Jones &amp; Sam Barrows, <a href="https://pydata.org/amsterdam2019/schedule/presentation/19/ubers-synthetic-control-experimentation-framework-when-ab-tests-are-not-possible/">“Uber’s Synthetic Control Experimentation Framework When A/B Tests Are Not Possible”</a>, <em>PyData Amsterdam</em>, 2019. Network effects, whole-market treatment, and donor-city synthetic controls. <a href="https://www.youtube.com/watch?v=j5DoJV5S2Ao">Talk video</a>.</li><li>Alan Gerber &amp; Donald Green, <em>Field Experiments</em>, W. W. Norton, 2012.</li><li>Joseph Henrich, Steven Heine &amp; Ara Norenzayan, <a href="https://doi.org/10.1017/S0140525X0999152X">“The Weirdest People in the World?”</a>, <em>Behavioral and Brain Sciences</em> 33(2–3), 2010.</li><li>Joshua Angrist &amp; Jörn-Steffen Pischke, <em>Mostly Harmless Econometrics</em>, Princeton University Press, 2009.</li><li>Course lecture: <a href="https://youtu.be/fjBFe8cRY8c">Natural Experiments</a>.</li><li>Course lecture: <a href="https://youtu.be/Fxx5l5Gh2Yk">Lab Experiments</a>.</li><li>Course lecture: <a href="https://youtu.be/PMyy9AGSu54">Field Experiments</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=458159b78e15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: Your Metric Is a Business Decision]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-your-metric-is-a-business-decision-17bdb126ac0a?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/17bdb126ac0a</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 05:01:02 GMT</pubDate>
            <atom:updated>2026-10-09T05:01:02.225Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>What happens when the metric improves and the product outcome gets worse?</em></h4><p><em>Part 4 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LyJEIT6-1HjKBe83R18f1w.png" /></figure><p>A feature increased engagement.</p><p>Users explored more profiles.</p><p>If that were the metric on the experiment dashboard, the result would look encouraging.</p><p>There was only one problem.</p><p>They got fewer matches.</p><p>That happened in a randomized field experiment on a major North American online dating platform.</p><p>The researchers selected 100,000 new users at random. Half received one month of free access to anonymous browsing, allowing them to visit profiles without leaving the usual visible trace.</p><p>Anonymous users browsed more.</p><p>But a profile visit was doing something the engagement metric did not price.</p><p>It created a weak signal of interest.</p><p>Someone could see that another person had visited their profile and respond.</p><p>Anonymous browsing removed that signal.</p><p>The result was a product lesson disguised as a measurement problem:</p><p><strong>the proxy improved while the outcome the product existed to create deteriorated.</strong></p><p>That is why choosing a metric is never merely an analytics task.</p><p>It is a decision about what counts as success.</p><h3>The metric gets a vote</h3><p>Imagine two teams reviewing that same experiment.</p><p>Team A has decided that success means increasing <strong>profiles viewed</strong>.</p><p>Team B has decided that success means increasing <strong>matches</strong>.</p><p>They can observe the same treatment, the same users, and the same data and still reach different decisions.</p><p>The disagreement is not statistical.</p><p>The teams have given different outcomes authority over the product decision.</p><p>That authority needs to be assigned before the experiment runs.</p><p>Otherwise every dashboard becomes an invitation to find whichever metric supports the story we already want to tell.</p><p>This is the role of an <strong>Overall Evaluation Criterion</strong>, or OEC.</p><p>An OEC is the pre-specified evaluation criterion used to judge whether an experiment achieved its objective.</p><p>Sometimes one primary metric can do that job.</p><p>Sometimes several objectives need to be combined.</p><p>The implementation can vary.</p><p>The important part is that the decision logic exists before the outcome is known.</p><h3>Metrics form a hierarchy, not a pile</h3><p>Experiment dashboards often place many numbers next to one another.</p><p>That visual equality is misleading.</p><p>The numbers play different roles.</p><p>A <strong>primary metric</strong> represents the main outcome the experiment is designed to change.</p><p><strong>Secondary metrics</strong> help diagnose the mechanism.</p><p>A <strong>guardrail</strong> protects something the team is unwilling to sacrifice while improving the primary outcome.</p><p>A <strong>North Star metric</strong> may align an organization over months or years while being too slow or broad for a short experiment.</p><p>And a <strong>proxy</strong> stands in for an outcome that matters but cannot be observed quickly enough for the current decision.</p><p>Those are not interchangeable categories.</p><p>If subscription renewal takes 60 days, a two-week experiment may need an earlier signal.</p><p>That does not transform the early signal into renewal.</p><p>It creates a measurement assumption:</p><blockquote><em>this short-term behavior tells us something reliable about the long-term outcome.</em></blockquote><p>The assumption is now part of the experiment.</p><h3>Delayed outcomes turn measurement into a design problem</h3><p>Recommendation systems run into this problem constantly.</p><p>A click arrives now.</p><p>Long-term satisfaction does not.</p><p>A show can be recommended today and become a habit several weeks later.</p><p>Spotify researchers studying podcast recommendations faced exactly this kind of delayed reward.</p><p>The long-term outcome involved repeated engagement over roughly two months.</p><p>Waiting until the full reward matured before updating the policy would make learning painfully slow.</p><p>Optimizing a quick proxy would accelerate learning but risk optimizing the wrong behavior.</p><p>Their approach used partial outcomes as they arrived to update a probabilistic model of the eventual long-term reward.</p><p>The system did not pretend the early metric <em>was</em> the long-term outcome.</p><p>It used the early evidence to learn about it.</p><p>That distinction is the useful lesson.</p><p>If your real outcome arrives outside the experiment window, you have a few choices:</p><p>wait,</p><p>use a validated proxy,</p><p>model the delayed reward,</p><p>or create another defensible bridge.</p><p>What you cannot do safely is replace the long-term outcome with whatever is easiest to measure and silently treat them as equivalent.</p><h3>Funnel metrics are business choices too</h3><p>Suppose an e-commerce team wants to improve performance.</p><p>Where should success live?</p><p>Product-page click-through?</p><p>Add-to-cart rate?</p><p>Checkout completion?</p><p>Revenue?</p><p>Each is defensible for a different decision.</p><p>Optimizing higher in the funnel gives quicker and often more sensitive signals.</p><p>It also moves the measurement farther away from the business outcome.</p><p>Moving lower in the funnel may bring the metric closer to value while making experiments slower and noisier.</p><p>Even a familiar metric such as CTR contains hidden choices.</p><p>Clicks per impression?</p><p>Users who clicked at least once?</p><p>Clicks per session?</p><p>The events may be identical.</p><p>The metric is not.</p><p>Session definitions create the same problem.</p><p>A 15-minute inactivity rule may make sense for one product and badly distort a purchase cycle that unfolds over several days.</p><p>The analytics tool may supply a default.</p><p>That does not make the default the correct experimental unit.</p><h3>A testable hypothesis earns you the right to choose a metric</h3><p>Before deciding which metric should move, you need a claim precise enough to fail.</p><p>Compare these two hypotheses:</p><blockquote><em>A higher initial price anchor on the product page will increase average transaction value.</em></blockquote><p>and:</p><blockquote><em>A nicer design will make users happier.</em></blockquote><p>The first identifies an intervention, an outcome, and a direction.</p><p>The second leaves almost every important word undefined.</p><p>What exactly changes?</p><p>What does happier mean?</p><p>Which observable outcome would count as failure?</p><p>A hypothesis should constrain the experiment.</p><p>Otherwise the metric selection process becomes another place to retrofit the story after the result arrives.</p><p>Observational tools can help generate hypotheses.</p><p>A reporting platform may reveal that a creative pattern appears to perform differently across industries or weeks.</p><p>Useful.</p><p>Interesting.</p><p>Worth testing.</p><p>But an observed pattern is still not a stable product law, and it is not automatically causal.</p><p>Hypothesis generation and hypothesis testing are different jobs.</p><h3>Guardrails are where trade-offs become real</h3><p>Now imagine the primary metric improves by 3%.</p><p>At the same time:</p><p>p95 latency increases by 400 milliseconds.</p><p>Cost per request doubles.</p><p>Or unsubscribe rate crosses a threshold the team had agreed was unacceptable.</p><p>Did the treatment win?</p><p>You cannot answer that from the primary metric alone.</p><p>A guardrail is useful precisely because it can constrain a decision that otherwise looks positive.</p><p>That does not mean every guardrail has to be a hard veto.</p><p>A breach might trigger escalation, another experiment, a narrower rollout, or additional analysis.</p><p>But the role should be known before the result.</p><p>A metric discovered after a disappointing launch may still be important.</p><p>It just was not a guardrail for that experiment.</p><h3>Composite metrics hide policy inside arithmetic</h3><p>Sometimes a single metric genuinely cannot represent the decision.</p><p>A team may care about engagement, conversion, revenue, retention, and cost simultaneously.</p><p>So it constructs a composite OEC.</p><p>Now the uncomfortable question arrives:</p><p>How much is one objective worth relative to another?</p><p>Change the weights and the winning variant may change.</p><p>That is not a mathematical bug.</p><p>The weights encode the organization’s trade-offs.</p><p>Historical data can help estimate relationships between metrics.</p><p>Machine-learning models can predict downstream outcomes.</p><p>Optimization can explore possible frontiers.</p><p>But none of those techniques removes the need for someone to own the business choice.</p><p>If a model silently determines that a 2% retention loss is acceptable in exchange for a 5% engagement gain, the organization has not eliminated a policy decision.</p><p>It has hidden one.</p><h3>Return to the dating experiment</h3><p>The anonymous-browsing result is useful because it exposes all of these issues at once.</p><p>The treatment increased one kind of engagement.</p><p>It also removed a weak social signal.</p><p>That mechanism mattered for matching.</p><p>The reduction in matching was especially pronounced for women in the platform context studied by the researchers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IeB76tSuOZn09uLqfFyvhQ.png" /></figure><p>The correct takeaway is not that anonymous browsing is universally harmful.</p><p>This was one platform, one period, and one social context.</p><p>The transferable lesson is narrower and more useful:</p><p><strong>a metric can move in the desired direction while the mechanism that creates product value becomes weaker.</strong></p><h3>The experiment begins before assignment</h3><p>When teams talk about experiment design, they often start with randomization.</p><p>But some of the most consequential decisions have already happened before anyone is assigned to treatment.</p><p>What is the hypothesis?</p><p>What is the outcome?</p><p>What is the unit?</p><p>What is the time window?</p><p>Which metric gets decision authority?</p><p>Which dimensions are protected by guardrails?</p><p>What happens when the metrics disagree?</p><p>These choices determine what evidence the experiment will eventually be capable of producing.</p><p>Statistics can quantify the difference between treatment and control.</p><p>They cannot decide what your company should value.</p><h3>Explore the trade-off in code</h3><p><a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/solutions/unit-04-metrics-demo.ipynb">This notebook</a> sends two homepage variants through the same funnel and evaluates the observed data under two different OEC definitions.</p><p>Change the weights and the winner changes.</p><p>It also counts the same click events under user-level and session-level definitions, showing how the metric itself can change before any hypothesis test is run.</p><p>For readers who want to build the analysis themselves, <a href="https://colab.research.google.com/github/dinbav/digital-experiments-course/blob/main/notebooks/unit-04-metrics-exercise.ipynb">the companion notebook</a> provides the same setup with the analysis left open.</p><h3>Questions to think about</h3><ul><li>Which metric in your product could improve while the product itself becomes worse?</li><li>Which long-term outcome are you currently replacing with a short-term proxy?</li><li>Would one of your guardrails actually overturn an otherwise positive experiment?</li><li>Who owns the weights when two objectives conflict?</li></ul><h3>Sources and further reading</h3><ul><li><a href="https://doi.org/10.1287/mnsc.2015.2301">Bapna, Ramaprasad, Shmueli &amp; Umyarov, “One-Way Mirrors in Online Dating: A Randomized Field Experiment,” <em>Management Science</em> 62(11), 2016</a>. A randomized field experiment in which 50,000 of 100,000 new users received anonymous browsing: profile viewing increased, matching outcomes decreased, and the effect was particularly strong for women.</li><li>Kohavi, R., Tang, D., &amp; Xu, Y., <em>Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing</em>. Cambridge University Press, 2020. Chapters 6–7 discuss evaluation criteria, metric hierarchies, and guardrails.</li><li><a href="https://doi.org/10.1145/3580305.3599386">McDonald, Maystre, Lalmas, Russo &amp; Ciosek, “Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay,” KDD 2023</a>. A method for learning about long-term rewards from progressively revealed partial outcomes. <a href="https://arxiv.org/abs/2307.09943">Open preprint</a>.</li><li>Course lecture: <a href="https://youtu.be/0UTn97HFY48">Evaluation Metric Definition</a>. Consumption funnels and user/session/window definitions.</li><li>Course lecture: <a href="https://youtu.be/hWIvcALwosE">Terminology and Notation</a>. OEC terminology used throughout the course.</li><li>Course lectures: <a href="https://youtu.be/5krIKyN4Qwk">Hypothesis</a> and <a href="https://youtu.be/KCcYqqw8SuA">Elements of a Good Business Hypothesis</a>. What makes a business hypothesis testable.</li><li>Course lecture: <a href="https://youtu.be/-bT2GO4ojgM">Taboola Trends</a>, with the live <a href="https://trends.taboola.com/">Taboola Trends</a> tool. The creative patterns shown in the lecture are tool demonstrations used for hypothesis generation, not findings from a published study.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17bdb126ac0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Digital Experiments: The Outcome You Never Get to See]]></title>
            <link>https://medium.com/@dinabavli/digital-experiments-the-outcome-you-never-get-to-see-dcee2bca8f07?source=rss-ae5cca1dfa71------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcee2bca8f07</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[Dina Bavli]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 05:01:02 GMT</pubDate>
            <atom:updated>2026-10-06T05:01:02.033Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Potential outcomes, counterfactuals, and why causal inference begins with a value that is missing by construction.</em></h4><p><em>Part 3 of 12 in the Digital Experiments series</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*144OlYlUPvaz7SGFsEBfIQ.png" /></figure><p>For every person in an experiment, there are two answers we would like to know.</p><p>What would happen if they received the treatment?</p><p>And what would happen if they did not?</p><p>We will observe exactly one.</p><p>The other is not hidden somewhere in the logs. It is not waiting for better instrumentation, a larger warehouse, or a more sophisticated model.</p><p>It is the outcome of a world that did not happen.</p><p>That missing value is the reason causal inference needs a framework at all.</p><h3>Start with one student and two possible worlds</h3><p>Suppose an online course has poor completion.</p><p>Students are randomly assigned to one of two reminder strategies:</p><ul><li>generic weekly reminders</li><li>personalized weekly reminders</li></ul><p>The outcome is weeks of participation.</p><p>A small teaching example might produce group averages of 6.83 and 6.67 weeks, roughly one-sixth of a week apart.</p><p>It would be easy to call that difference “the treatment effect.”</p><p>But that skips the most important step.</p><p>For student (i), define two <strong>potential outcomes</strong>:</p><p><strong>Yᵢ(1)</strong> = the number of weeks that student would participate under the personalized reminder</p><p><strong>Yᵢ(0)</strong> = the number of weeks that same student would participate under the generic reminder</p><p>Let <strong>Dᵢ</strong> indicate the assignment:</p><p><strong>Dᵢ = 1</strong> for treatment</p><p><strong>Dᵢ = 0</strong> for control</p><p>The outcome we actually observe is:</p><p><strong>Yᵢ = DᵢYᵢ(1) + (1-Dᵢ)Yᵢ(0)</strong></p><p>That equation is doing something very simple.</p><p>If the student is treated, we observe Yᵢ(1).</p><p>If the student is in control, we observe Yᵢ(0).</p><p>Never both.</p><p>The individual causal effect can be written as:</p><p><strong>τᵢ = Yᵢ(1) — Yᵢ(0)</strong></p><p>This is the most granular causal contrast we could define for that student.</p><p>And it requires the one thing the data cannot give us: both potential outcomes for the same unit under the same circumstances.</p><p>Paul Holland called this the <strong>fundamental problem of causal inference</strong>.</p><p>The individual effect is not merely difficult to measure.</p><p>Within the potential-outcomes framework, it is fundamentally unobservable as a direct comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CDjsSdZUASv9D93EtbtbnA.png" /></figure><h3>Why giving everyone both treatments does not reveal the missing cell</h3><p>The obvious response is:</p><p>Why not use a crossover design?</p><p>Give each participant A in one period and B in another. Then everyone serves as their own control.</p><p>Crossover designs are real, useful randomized designs. In the right setting, they can be more efficient than parallel-group experiments because comparisons occur within participants.</p><p>But they do not abolish the fundamental problem.</p><p>The outcome for a person receiving B in period 2 is not literally the outcome we would have observed had that same person received B instead of A in period 1.</p><p>Time has passed.</p><p>The person’s state may have changed.</p><p>They may have learned something.</p><p>The first treatment may still affect the second period.</p><p>The outcome itself may vary by period.</p><p>A well-designed crossover trial deals with these issues explicitly. Treatment sequences are randomized. Period effects are considered. Washout periods may be used when appropriate. The design depends on assumptions about carryover and stability.</p><p>Under suitable conditions, crossover trials can estimate treatment effects very effectively.</p><p>What they do not do is let us observe both potential outcomes for one person at the same time under identical conditions.</p><p>The missing counterfactual remains missing.</p><h3>Estimand and estimator are not the same thing</h3><p>Once the individual effect is unobservable, we need to decide what causal quantity we actually need.</p><p>A common choice is the <strong>Average Treatment Effect</strong>:</p><p><strong>ATE = E[Y(1) — Y(0)]</strong></p><p>This is an <strong>estimand</strong>.</p><p>An estimand is the causal quantity the study is trying to learn.</p><p>An <strong>estimator</strong> is the rule we use to learn about that quantity from observed data.</p><p>In a randomized parallel-group experiment, a familiar estimator is the difference in observed group means:</p><p><strong>τ̂ = Ȳ₁ — Ȳ₀</strong></p><p>Those expressions are not interchangeable.</p><p>The first describes the causal target.</p><p>The second describes a calculation from data.</p><p>Randomization is what connects them.</p><p>Under the relevant design assumptions, assignment makes treatment and control groups comparable by design. Over the randomization, the difference in group means can provide an unbiased estimate of an average causal effect for the experimental sample. Extending that conclusion to a broader population introduces additional questions about sampling and external validity.</p><p>This distinction matters because teams often discuss “the metric” and “the effect” as if they were the same object.</p><p>They are not.</p><p>The outcome tells us what was observed.</p><p>The estimand tells us what causal quantity we want.</p><p>The estimator tells us how we intend to recover information about it.</p><h3>The average can hide opposing effects</h3><p>Suppose a treatment helps one group by five units and harms another group by five.</p><p>The average effect could be close to zero.</p><p>Does that mean the treatment does nothing?</p><p>Not necessarily.</p><p>It means the average may not be the only quantity relevant to the decision.</p><p>This is the motivation for <strong>heterogeneous treatment effects</strong>.</p><p>We may want to estimate effects for pre-specified subgroups or conditional average treatment effects as a function of observable characteristics.</p><p>Modern causal machine-learning methods can help with that.</p><p>But there is an important conceptual boundary.</p><p>Estimating that people with characteristics X tend to have a larger treatment effect is not the same as recovering the unobservable individual effect τᵢ for a particular person.</p><p>A conditional average asks something like:</p><blockquote><em>What is the average treatment effect among people like this?</em></blockquote><p>The impossible individual comparison asks:</p><blockquote><em>What would happen to this exact person in both mutually exclusive worlds?</em></blockquote><p>Those are different estimands.</p><p>“Does it work for people like me?” is already harder than “does it work on average?”</p><p>“Exactly what will it do to me?” is harder still.</p><h3>Notation forces the treatment to become concrete</h3><p>Potential-outcomes notation can look unnecessarily formal until you try to write it for a real product experiment.</p><p>Suppose:</p><p><strong>Yᵢ(1)</strong> = course participation if student (i) receives personalized reminders.</p><p>What exactly is:</p><p><strong>Yᵢ(0)</strong>?</p><p>No reminder?</p><p>A generic reminder?</p><p>The current production reminder?</p><p>The same message without personalization?</p><p>Those are different control conditions, and therefore different causal questions.</p><p>The notation forces us to specify them.</p><p>It does the same thing to the outcome.</p><p>In this example, Y was “weeks of participation.”</p><p>But perhaps the real product goal is course completion.</p><p>Or learning.</p><p>Or satisfaction.</p><p>Or revenue.</p><p>A perfectly executed experiment can answer the wrong causal question if we defined Y poorly.</p><p>That is why notation is not mathematical decoration.</p><p>It exposes product decisions that prose often hides.</p><h3>The same missing world exists outside randomized experiments</h3><p>Potential outcomes are not only a language for A/B tests.</p><p>Suppose a policy changes and randomization is impossible.</p><p>We still want to compare:</p><p><strong>Y(1):</strong> what happened under the policy</p><p>with</p><p><strong>Y(0):</strong> what would have happened without it.</p><p>The second world remains missing.</p><p>A quasi-experimental design does not make that counterfactual observable. It uses structure, comparison units, assignment rules, or assumptions to identify information about it.</p><p>Difference-in-differences, for example, may identify an average treatment effect on the treated in a canonical design under an appropriate parallel-trends assumption.</p><p>That assumption is itself counterfactual.</p><p>It says, in effect, that absent treatment, the treated group’s untreated potential outcomes would have evolved like those of the comparison group.</p><p>We never directly observe that untreated path after treatment begins.</p><p>That is why causal identification always has an assumption somewhere.</p><p>Randomization makes some assumptions unusually defensible.</p><p>Quasi-experimental designs replace random assignment with other identifying structure.</p><p>The counterfactual problem itself never disappears.</p><h3>Write the estimand before choosing the method</h3><p>There is a habit I want to carry through the rest of this series.</p><p>Before deciding how to analyze an experiment, complete this sentence:</p><blockquote><strong><em>The causal effect we want to estimate is…</em></strong></blockquote><p>Then write the two potential outcomes in words.</p><p>What exactly changes between Y(1) and Y(0)?</p><p>For whom?</p><p>Over what time window?</p><p>What outcome is measured?</p><p>Are we interested in an average effect across everyone, an effect among a pre-specified subgroup, an effect on those actually treated, or something else?</p><p>That exercise often reveals that a team has chosen an estimator before agreeing on the estimand.</p><p>A regression is not an estimand.</p><p>A t-test is not an estimand.</p><p>Difference-in-differences is not an estimand.</p><p>They are tools.</p><p>The causal quantity comes first.</p><h3>What it depends on</h3><p>Which effect you can credibly estimate depends on several pieces of the design.</p><p><strong>Treatment definition.</strong> Are Y(1) and Y(0) genuinely well-defined alternative experiences?</p><p><strong>Assignment mechanism.</strong> Why did some units receive treatment and others not?</p><p><strong>Estimand.</strong> Are you targeting an average effect, an effect for the treated, a subgroup effect, or another quantity?</p><p><strong>Stability.</strong> Can time, carryover, learning, interference, or changing treatment versions alter what those potential outcomes mean?</p><p><strong>Population.</strong> Is the target the experimental sample or a broader population?</p><p><strong>Measurement.</strong> Does Y represent the business outcome you actually care about?</p><p>That list is why writing down Y(1), Y(0), and the estimand before opening the analysis notebook is worth the inconvenience.</p><h3>Questions to think about</h3><ul><li>For a treatment your team is considering, write Y(1) and Y(0) in words. Are they genuinely different, well-defined experiences?</li><li>Which potential outcome is missing for every treated user? Which is missing for every control user?</li><li>Does your decision require an average effect, a subgroup effect, or are you implicitly asking for an individual effect?</li><li>If your ATE were zero, what pattern of heterogeneous effects would still make you change the product decision?</li></ul><h3>Sources and further reading</h3><ul><li><a href="https://doi.org/10.1080/01621459.1986.10478354">Paul W. Holland, “Statistics and Causal Inference,” <em>Journal of the American Statistical Association</em> 81(396), 1986, 945–960</a>. The paper that named the fundamental problem of causal inference.</li><li><a href="https://doi.org/10.1037/h0037350">Donald B. Rubin, “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies,” <em>Journal of Educational Psychology</em> 66(5), 1974</a>. A foundational paper for the potential-outcomes framework.</li><li>Gerber, A. S., &amp; Green, D. P., <em>Field Experiments: Design, Analysis, and Interpretation</em>. W. W. Norton, 2012. Chapter 2 develops potential outcomes and average treatment effects.</li><li>Imbens, G. W., &amp; Rubin, D. B., <em>Causal Inference for Statistics, Social, and Biomedical Sciences</em>. Cambridge University Press, 2015. A detailed treatment of potential outcomes, estimands, estimators, and assignment mechanisms.</li><li><a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC6667942/">CONSORT 2010 statement: extension to randomised crossover trials</a>. Methodological guidance on crossover designs, period effects, carryover, and analysis.</li><li>Course lecture: <a href="https://youtu.be/hWIvcALwosE">Terminology and Notation</a>. Potential outcomes and the Dᵢ assignment indicator used throughout this series.</li><li>Course lecture: <a href="https://youtu.be/K1RHeiUJzqY">Terminology and Notation — worked example</a>. The reminder example, the 6.83/6.67 teaching illustration, crossover discussion, and ATE.</li><li>Course lecture: <a href="https://youtu.be/dpnezDRjWtc">An Experimental Approach to Inferring Causality</a>. Connecting causal questions to the evidence needed to answer them.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcee2bca8f07" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>