<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Garret O&#39;Connell on Medium]]></title>
        <description><![CDATA[Stories by Garret O&#39;Connell on Medium]]></description>
        <link>https://medium.com/@garret.oconnell?source=rss-5320a27dd81------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gkzWSJlM0tiygmwruC6_yw.jpeg</url>
            <title>Stories by Garret O&amp;#39;Connell on Medium</title>
            <link>https://medium.com/@garret.oconnell?source=rss-5320a27dd81------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:21:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@garret.oconnell/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CUPED for switchback tests]]></title>
            <link>https://medium.com/@garret.oconnell/cuped-for-switchback-tests-9e5b924ce1b0?source=rss-5320a27dd81------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e5b924ce1b0</guid>
            <category><![CDATA[switchback]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[marketplaces]]></category>
            <dc:creator><![CDATA[Garret O'Connell]]></dc:creator>
            <pubDate>Wed, 04 Sep 2024 12:12:12 GMT</pubDate>
            <atom:updated>2024-09-04T13:02:24.757Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-k79oObQl50-zNW9R9IQBQ.jpeg" /></figure><h3>Authors</h3><p>This post was written by Garret O’Connell, with contributions from Carlos Bentes and Kenil Cheng. It was supported by the work of the Bolt Experimentation Team: Gabriela-Raluca Săsăran, Andrey Kuzmin, August Hovland, and Pavel Kiper.</p><h3>TL;DR</h3><p>We explored a two-stage CUPED variance reduction approach to speed up the time switchback tests take to reach power. We found major reductions in test duration in the range of 25–50%. We flagged some factors that appear to modulate this effect and proposed extensions for potential further test duration reductions.</p><h3>Background</h3><p>It’s been over a year since we at Bolt’s Experimentation Team presented some of our early learnings in running switchback tests (see <a href="https://medium.com/bolt-labs/tips-and-considerations-for-switchback-test-designs-d1bd7c493024">Tips and considerations for switchback test designs</a>) to account for network interference effects in 2-sided marketplaces. Since then, we’ve heard mainly one request from teams using switchback tests — <em>make them FASTER!</em></p><p>Switchback test designs are generally much slower than user- or session-randomised tests to reach similar power levels, as they will have smaller sample sizes — there are fewer timeslices than users. This can slow development for teams that depend on switchback tests because their products affect marketplace dynamics, which can harm the business as they often affect impactful products (e.g. pricing), which we want to keep tuned to as frequently as possible.</p><p>Variance reduction methods help speed up A/B testing to reach statistical power faster. CUPED (Controlled-experiment Using Pre-Experiment Data) is a popular variance reduction method. Still, we haven’t seen documented cases of its application in switchback tests (see <a href="https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d">here</a> for a beginner’s guide). There is an example from DoorDash using a related ML approach called CUPAC <a href="https://doordash.engineering/2020/06/08/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/">here</a>. Still, for companies that don’t have ML infrastructure in their data pipelines, CUPED could be more practically useful.</p><h3>Historical metric as a covariate</h3><p>CUPED usually takes user metric values from a pre-experiment period and uses this treatment-unrelated information to form an expected baseline for the experiment period. This removes some of the treatment-unrelated variance, allowing us to increase the sensitivity to or speed of detection of treatment effects.</p><p>One challenge in translating this approach to switchback test designs is that there is no “user” with historical data from which to build baselines, as switchbacks randomise over timeslices, which never repeat. However, we can consider timeslices from the same weekly periods (e.g., Mondays at 9–10 a.m.) as related in terms of the metric values we would expect. This allows us to build histories for timeslices and use this information about weekly seasonalities to explain away some of the treatment-unrelated variances.</p><p>Here, we used repeating user activity patterns over weekly periods as a CUPED covariate. As is recommended for CUPED, we used the pre-experiment values of the outcome metric as a covariate. This is the first of two stages of CUPED we applied. Let’s call this stage CUPED-metric, as it uses values from the same metric as a covariate.</p><h3>Experiment vs. historical delta of demands as a covariate</h3><p>CUPED can be used with multiple covariates, allowing us to remove treatment-unrelated variance further. We asked ourselves which other information could complement the usual CUPED-metric covariate.</p><p>A limitation of using only historical data is that it relies on past trends to explain activity during the experiment. If trends change from the past to the experiment period, CUPED’s ability to reduce variance is weakened. To handle these changes, we could include a covariate that tracks changes between pre-experiment and experiment periods.</p><p>Before continuing, some of you might be concerned that using information from during the experiment could remove variance that’s affected by (or belongs to) the treatment and, therefore, risk underestimating the treatment variance and biased results. However, the original CUPED paper (<a href="https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf">ref</a>) explicitly suggests using within-experiment covariates when the covariate is unaffected by the treatment.</p><p>One example of such a covariate metric is the number of users with sessions (i.e., opened the app) in a given time slice, an indicator of demand during that time. This metric is unlikely to be affected by any feature exposed during the session because users would only see such features after opening the app and logging a session, increasing the metric.</p><p>As a second CUPED covariate, we included the delta of this demand metric — the number of users logged in a period — between pre- and within-experiment periods. This allowed us to complement our purely historical information from the CUPED metric with an indicator of recent shifts for a given period. Let’s call this stage CUPED-demand.</p><h3>Cuped calculation steps</h3><p>Below is an overview of the steps for how CUPED-metric and CUPED-demand can be implemented in a simple ways. In this example, we use revenue as the outcome metric. Timeperiod denotes an identifier of “dayofweek — hour” of when the user started their first session within a timeslice.</p><p>The basic steps were as follows:</p><ol><li>Pull unit-level test data for test and pre-test windows</li><li>Join test and pre-test tables and calculate inputs to CUPED parameters: average values over pre-test periods for the metric and demand, average metric values over test timeslices for the metric and demand, delta of within timeslice vs. pre-experiment demand</li><li>Calculate theta and covariate averages over periods</li><li>Calculate CUPED for the metric using historical data of the metric</li><li>Repeat steps 2–4 using the CUPED-metric as the metric and the delta demand as the covariate to calculate CUPED-demand</li></ol><h3>CUPED evaluation approach</h3><p>To evaluate how much faster tests could reach the target power using CUPED metrics, we calculated the required sample size to reach 80% power for a Minimum Detectable Effect (MDE) of 2% at an alpha of 0.05 (5%). Since the required sample size proportionally increases with test duration for switchback tests, we used the size of sample needed to track CUPED’s impact on test speed.</p><p>Switchback tests are examples of “clustered” data where observations within a timeslice cluster can share characteristics. This needs to be corrected when estimating variance in statistical tests (see previous <a href="https://medium.com/bolt-labs/tips-and-considerations-for-switchback-test-designs-d1bd7c493024">post</a> for details). To do this a Variance Inflation Factor (VIF) which is a correction factor that can be applied to required sample sizes or standard errors to account for the correlation between units within timeslices. Using the VIF is analogous to using Mixed Linear Models with timeslice as a random factor (see <a href="https://doordash.engineering/2019/09/11/cluster-robust-standard-error-in-switchback-experiments/">here</a> for an example).</p><h3>A/A test data set</h3><p>We evaluated the required sample sizes in one large city (~3.5 M residents) using two weeks of user data each for pre- and within-experiment windows. One-hour timeslices were randomised for treatment and control using an alternating schedule. Weekly periods were defined as one hour. The outcome metric was the sum of revenue per user per timeslice, and the demand metric was the number of users with sessions per timeslice. Evaluations compared required sample sizes between metric versions of non-CUPED, CUPED-metric, and CUPED-demand.</p><h3>CUPED evaluation results</h3><p>Below are the results of evaluating each CUPED stage on the switchback A/A test user data. We didn’t show the absolute numbers for confidentiality purposes but instead expressed the impact relative to the non-CUPED metric version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fTh-t-tO_h30nyrJ.png" /></figure><p>After applying both CUPED stages, we cut our test duration by over half. Most cities we investigated showed reductions in test duration in the 25–40% range. Like any CUPED approach, it works better for some metrics than others, but it can be applied equally to continuous, count, or rate metrics. We could think of ways to add more information about these weekly seasonalities that could explain variance over timeslices. For example, the above CUPED procedure could be applied separately for different geographical regions of a city.</p><p>We included results of the impact on the variance between and within timeslices to illustrate one particular point of how CUPED for switchbacks works. The results show that CUPED does nothing for the variance within a timeslice (between users) but reduces the variance between timeslices. This is because we used timeslice-level information in the CUPED approach, not user-level.</p><p>When we reduce the variance between timeslices, we’re also reducing the extent to which our data is clustered, i.e. if timeslices are more similar, then relative similarities between users in the same timeslices decrease. This means we don’t have to correct as much for clustering in the VIF, as seen in how CUPED decreases the VIF. CUPED over periods thus works by “de-correlating” units within each timeslice (the same point is made <a href="https://doordash.engineering/2019/02/20/experiment-rigor-for-switchback-experiment-analysis/">here</a> in the Appendix section).</p><p>We’ve noted that for some cities, the length of the period or timeslice can be influential. For example, below are the results for city B for different lengths of timeslice. Note: The baseline for each timeslice length is different from the non-CUPED metric version for that length.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AHelFe3SNp7_epcK.png" /></figure><p>We see that the gains were around double those for longer timeslices than for the shorter timeslices. Tests with longer timeslices can, therefore, enjoy a larger relative impact. Note that this does not mean these tests will run faster than those with shorter timeslices, as the baselines of the above results are specific to the timeslice length. Tests with longer timeslices have smaller sample sizes so that they will run longer. In the above results, the baseline test duration for 3-hour timeslices was 2 times longer than that for 1 hour.</p><h3>Summary</h3><p>Here, we presented a two-stage method of variance reduction using CUPED for switchback tests. This could provide a much-needed boost in development speed for many teams, as switchbacks tend to run impractically long. While more general methods exist (see <a href="https://doordash.engineering/2020/06/08/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/">here</a>), the presented approach can be implemented purely in SQL, lowering the bar for implementation.</p><h3>Further reading</h3><ul><li>Data Council <a href="https://www.datacouncil.ai/talks24/case-studies-from-a-methodologist-on-an-experimentation-platform?hsLang=en">talk</a> by Laura Cosgrove recommending the use of demand metrics as a CUPED covariate</li><li>Research <a href="https://www.researchgate.net/profile/Yixin-Tang-5/publication/345698207_Control_Using_Predictions_as_Covariates_in_Switchback_Experiments/links/5fab109b458515078107aa8b/Control-Using-Predictions-as-Covariates-in-Switchback-Experiments.pdf">paper</a> on using CUPAC for reducing variance in switchback test designs</li></ul><h3>Appendix</h3><p>In addition to the above covariates, we also explored variations on the set of covariates, e.g., using only the historical or within-experiment demand values as a covariate (and not the delta), and applying the CUPED-demand stage alone without the CUPED-metric stage. However, these were only somewhat impactful.</p><p>We didn’t observe any change in the average metric lift between treatment and control from non-CUPED to CUPED metric versions &gt; .001%, indicating no bias was introduced.</p><p>To further check for bias, we generated a permuted p-value distribution from 500 runs fully resampled with replacement data sets of the CUPED-demand version. As seen below, these were uniformly distributed, indicating no bias in terms of how the variance of the metrics is estimated. P-values were calculated at the user level and corrected for within-timeslice correlation between users by applying the VIF to the standard error of the treatment effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d7Bw2dTcnZ2fjuqY.png" /></figure><h3>Join us!</h3><p><a href="https://bolt.eu/">Bolt</a> is a place where you can grow professionally at lightning speed and create a real impact on a global scale.</p><p><a href="https://bolt.eu/en/careers/">Take a look at our careers page</a> and browse through hundreds of open roles, each offering an exciting opportunity to contribute to making cities for people, not cars.</p><p>If you’re ready to work in an exciting, dynamic, fast-paced industry and are not afraid of a challenge, we’re waiting for you!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q9DhK8sNmUt6jmlQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e5b924ce1b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tips and considerations for switchback test designs]]></title>
            <link>https://medium.com/bolt-labs/tips-and-considerations-for-switchback-test-designs-d1bd7c493024?source=rss-5320a27dd81------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1bd7c493024</guid>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[switchback]]></category>
            <category><![CDATA[network-effect]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Garret O'Connell]]></dc:creator>
            <pubDate>Thu, 24 Nov 2022 10:34:06 GMT</pubDate>
            <atom:updated>2022-11-24T14:36:49.104Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p8-dxx7ky7PaIxCCbRXEwQ.jpeg" /></figure><p>Co-written by Garret O’Connell and Carlos Bentes, data scientists in experimentation at Bolt</p><p>Switchback is a standard test design for conducting A/B tests in the industry where there is likely interference between units, e.g., two-sided marketplaces like ride-hailing services between riders and drivers or e-commerce platforms between sellers and buyers.</p><p>In these contexts, the treatment of one rider/buyer can affect the outcome of another, most obviously through changing the availability of drivers/sellers.</p><p>There are great introductions and analyses of switchback tests in articles <a href="https://medium.com/@DoorDash/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash-f1d938ab7c2a">here</a> and <a href="https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-b39db027a66e#.djox1933t">here</a>. Here, we want to add to the conversation by flagging issues we’ve encountered when using switchbacks that are applicable in many cases. So, if you’re thinking about conducting switchback tests — or already are — here’s a quick checklist of topics we’ll cover in this article:</p><ul><li>Tips on sanity checks for analysis of switchback data;</li><li>Tracking switchback treatment group assignments without data logging;</li><li>Running parallel tests without bias or conflicts;</li><li>Checking and handling bias due to inter-dependence between units of analysis;</li><li>Checking and handling bias due to imbalances in daily peak periods across treatment groups.</li></ul><h3>Switchbacks at Bolt</h3><p>Bolt offers various services like ride-hailing, food delivery, scooters, and car sharing.</p><p>To optimise the user experience of these services, Bolt has a centralised A/B testing platform that conducts dozens of tests per day across different units of randomisation (e.g. user, driver, courier) and automatically estimates the effects of feature changes on a wide range of performance metrics.</p><p>Switchbacks are used to test features that cause changes in the marketplace in terms of supply and demand — such as changes in dispatching logic of drivers/couriers. These features can’t be tested with the usual user/driver-level randomisation because the treatment group of one unit could likely affect the availability of services for units in the other treatment group, thereby causing interference bias.</p><h3>Sanity checks</h3><p>In switchback tests, the units of randomisation are timeslices. While this helps reduce bias due to interference between the service users (e.g. riders, drivers, couriers), analysing at the level of timeslices leads to a heavy loss in power. It’s also generally less easily interpretable in terms of business impact, e.g. knowing that there was a 2% increase in GMV per timeslice is difficult to translate to the average user experience.</p><p>For this reason, in switchback tests, the unit of randomisation often differs from the unit of analysis, e.g. the rows on which we want to calculate average outcomes. Often, we want to analyse the data at a lower granularity that maps onto units at the user level, e.g. riders/drivers.</p><p>However, users within the same timeslice will have correlated responses because their outcomes affect each other. This introduces the problem of correlation between units, which violates the assumption of independence of units in regression models used to estimate treatment effects. To counter this, we use cluster-robust standard errors (CRSE).</p><p>An overview of this issue has been presented in this <a href="https://doordash.engineering/2019/02/20/experiment-rigor-for-switchback-experiment-analysis/">article</a>. As a sanity check for this correction, we simulated A/A tests to get p-value distributions as a sanity check for the CRSE solution.</p><p>An A/A test is constructed by configuring both treatment and control with the same default condition. We can simulate multiple A/A switchback tests using historical data and randomly reassigning each timeslice between treatment and control. By construction, we expect the null hypothesis to hold (treatment and control are the same) and the p-values of our simulation to be uniformly distributed. Deviation from the uniform distribution indicates that a metric definition has bias.</p><p>Below, we can see the p-value distribution without any correction (left) and when CRSEs were used (right). This approach was easy to implement (a single argument in statsmodels regression) and appeared to provide an unbiased distribution of p-values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HsnFOrzWjwNyF65d17OTgA.png" /><figcaption>P value distributions for uncorrected (left) and CRSE-corrected (right) A/A tests.</figcaption></figure><h3>Assignment tracking</h3><p>Tracking the assignment of units to treatment variants in A/B testing adds engineering effort and complexity to data pipelines. We found a clever solution for switchbacks that allows us to ignore the storage of assignments altogether!</p><p>Hashing functions can create random but reproducible alphanumeric strings. Suppose we use hashing functions on some easily known properties of an observed unit in a city where a switchback test is running (e.g. a string of the test id and the timestamp of enrollment rounded down to the nearest timeslice start). We can get a treatment assignment simply by converting this string to bytes and reading the first few bytes. This trick makes treatment assignment for switchback “stateless” and a cinch to analyse.</p><h3>Sample size requirements</h3><p>Like any test, switchback tests need calculations for sample size requirements to ensure the test is sufficiently powered to detect treatment effects. Note that timeslice-randomised tests are basically “cluster” designs, where you randomise across groups of units that share some characteristics that might be relevant for treatment outcomes, e.g. the time of the day they’re observed. There is already rich literature on cluster-randomised sample size calculations to draw on (<a href="https://bmcmedresmethodol.biomedcentral.com/track/pdf/10.1186/1471-2288-11-102.pdf">1</a>,<a href="https://link.springer.com/article/10.1186/1471-2288-9-39"> 2</a>,<a href="https://sci-hub.wf/10.1016/j.cct.2015.02.011"> 3</a>,<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4521133/pdf/dyv113.pdf"> 4</a>).</p><p>This literature has derived a formula for calculating the power of cluster-randomised tests. It calculates an “inflation factor” (IF) that multiplies the sample size requirement if the test was randomised at the standard unit level of users, i.e. what our timeslice clusters consist of. For example, for timeslices of varying size (the typical case of having traffic that varies throughout the day), we use the following formula:</p><p><em>IF = 1 + ((cv2 + 1)m — 1)ρ</em></p><p>In this formula:</p><ul><li><em>m</em> is the average cluster size and number of units per timeslice;</li><li><em>cv </em>is the standard deviation of cluster size;</li><li><em>ρ</em> is the intra-cluster correlation coefficient (ICC) which can be obtained by running a mixed-effects model with clusters (timeslices) as a random factor and then dividing the within-group variation by the total variation.</li></ul><p>Here’s an example using the statsmodels package (timeslice_id is a unique timeslice identifier, and y is the outcome metric):</p><pre>res = smf.mixedlm(&quot;y~1&quot;, data, groups=data[&#39;timeslice_id&#39;]).fit()<br>within_variance = res.cov_re.values[0][0]<br>between_variance = res.scale<br>icc = within_variance/(within_variance+between_variance)</pre><p>We then multiply this IF on our standard unit-level sample size calculation. We use means and standard deviations from the most recent two weeks of metric data. From this, we can see how many days we’d need to run the test. We usually round this up to the nearest two-week interval to ensure the treatment groups are balanced for weekly seasonality.</p><h3>Inter-dependence between units-of-analysis</h3><p>Initially, we used “sessions” as the unit of analysis for timeslice-randomised tests, defined as an app open followed by an order or no order within 30 min of the last app activity. It was a sensible unit of analysis as sessions can be easily interpreted in terms of business impact (e.g. our feature causes X% lift on average per session). It also can be cleanly assigned to one treatment group, in contrast to users that can make orders in multiple timeslices and, therefore, could end up having multiple treatment group assignments.</p><p>However, we later discovered a serious bias risk with this unit of analysis choice.</p><p>One day, some stakeholders were testing a feature that caused sessions to crash and forced users to create a replacement session to order a ride. Notably, nearly all users who successfully replaced their crashed sessions with a new session made an order, as shown below.</p><p>This behaviour caused a reduction in the average order value per session because the crashed sessions had no value and brought down the average. Note that since orders were replaced, it didn’t lead to a loss in the sum value of orders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n9aZ9auLqfNJJQmeICfCTg.png" /><figcaption>Example scenario where a feature induced dependence between units (sessions) in how it caused users to replace crashed sessions.</figcaption></figure><p>Stakeholders could have easily interpreted this drop in average session value as indicating that the treatment led to a loss. While it’s true that a feature that causes crashes isn’t positive, we want stakeholders to get an accurate picture of treatment effects, so this misinterpretation isn’t ideal. Thankfully, our stakeholders spotted this, and we sought a solution to avoid this possible misinterpretation.</p><p>The problem is that this feature induced a dependence between sessions, because users were creating new sessions. This causes the biased interpretation of average changes in the session-level unit of analysis. The increased dependency between units also violates the assumptions of statistical tests used to estimate treatment effects.</p><p>How do we avoid these nasty biases? Change the unit-of-analysis to user would remove this dependence between units, but as mentioned, users can exist in multiple timeslices or treatment groups. So, we found a unit between sessions and users — “user-per-timeslice”, i.e. the unique user observed during a specific timeslice.</p><p>This unit controls for the dependence between sessions, at least those close to each other in the same timeslice, and is permanently assigned to only one treatment group. It’s also reasonably easily interpretable regarding the user experience relative to session-level changes.</p><p>To validate if this user-per-timeslice unit-of-analysis addresses the above biases, we looked at how it affected them in the above test data. Specifically, we wanted to see if this unit of analysis could:</p><ol><li>Reduce the imbalance in the number of units between treatment groups (which was high for session-level due to replacement sessions in the treatment group);</li><li>Reduce differences between treatment groups on unit-average metrics, i.e. eliminate the lower average order value seen when using session level.</li></ol><p>We performed these checks in the affected A/B tests and found that the imbalance in units between treatment groups went from -5.4% significant in session units to -0.6% non-significant in user-timeslice units (note: some degree of imbalance is expected in timeslice-randomised tests given differences in activity over periods).</p><p>We also found that differences in metrics between treatment groups went down, on average, by 5%. Sometimes the difference between “session” and “user-per-timeslice” units of analysis led to changes in the sign of significant effects (e.g. from positive to negative), indicating that the bias induced by this inter-dependence of units had a meaningful impact on interpretations of results.</p><h3>Imbalanced assignment of time periods</h3><p>Originally, we fully randomised the assignment of timeslices to treatment groups in switchback tests, i.e. each day had a completely random schedule. However, we noticed that this had the unfavourable effect of imbalances in the daily time period, e.g. busy lunchtimes between 12 p.m. and 2 p.m. being assigned more to one treatment group than the other.</p><p>To ensure daily periods were balanced, we developed a more constrained form of randomisation in which consecutive pairs of days were balanced by inverting the first day (i.e. switching assignments of treatment groups) to get the second day’s schedule. Then for successive weeks, the first week was fully inverted to ensure that if Monday lunchtime were treatment in the first week, it would be controlled in the second week (see illustration below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xL4srIroufwm61y49xJMcw.png" /><figcaption>Illustration of quasi-randomized switchback schedule over 2 week period.</figcaption></figure><p>Using this “quasi-randomised” switchback design, we achieved much higher levels of balance, which reduced the rate of significant false positives in A/A tests arising from this imbalance, e.g. users with a higher order value coming during lunchtimes being over-represented in the treatment vs the control group. <a href="https://code.tubitv.com/switchback-experimentation-at-tubi-d063645be030">Here a similar implementation is outlined</a>.</p><h3>Coordination of tests</h3><p>As teams began ramping up their testing, they wanted to experiment more with features that could affect the marketplace. This led to a surge in demand for switchback tests and put pressure on the testing backlog. We only allowed one switchback test per city to ensure tests were not interfering with each other, which can happen for simultaneously running switchback designs, e.g. treatment timeslices of one test occurring more often during the treatment timeslices of another concurrent test, leading to a bias in the estimation of these treatment effects.</p><p>To relieve this congestion, we developed a simple approach for coordinating simultaneous switchback tests with reduced risk of bias. To explain, we want to run two tests in parallel — test A and test B. Both options require that one test has timeslice durations that fit evenly into the other, e.g. if test A is two hours long, then test B can be an hour, 30 mins, or 10 mins, but it can’t be 45 mins.</p><p>Now, we simply ensure that test B has an alternating schedule that starts on one of test A’s switches (see below). True, it’s not ideal as alternating schedules have a higher number of annoying treatment switches (reducing this is why we do switchbacks in the first place), but it’s an acceptable tradeoff in some cases, especially given that it requires little added engineering effort. This way, we ensure that each timeslice of test A has both a control and a treatment of test B. Indeed, if test A is two hours and test B is one hour, there is still space for test C with 30 mins (if stakeholders are in a rush!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fPDWeF8OKwLtV3ks2Ou_8Q.png" /><figcaption>Illustration of parallel switchback test schedule over 2 day period.</figcaption></figure><p>We hope you find some value in our explanation and thoughts about switchback designs. Feel free to reach out to <a href="/service/mailto:garret.oconnell@bolt.eu">garret.oconnell@bolt.eu</a> with any questions.</p><h3>Join Bolt!</h3><p>We offer a unique opportunity for individuals to learn and develop while making a meaningful impact on millions of people across the globe in a hyper-growth environment.</p><p>If you feel inspired by our approach and want to join our journey, <a href="https://careers.bolt.eu/">check out our vacancies</a>. We have hundreds of open roles, including Data and Backend Engineer positions to help us work on our experimentation platform.</p><p>If you’re ready to work in an exciting, dynamic, fast-paced industry and are not afraid of a challenge, we’re waiting for you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1bd7c493024" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bolt-labs/tips-and-considerations-for-switchback-test-designs-d1bd7c493024">Tips and considerations for switchback test designs</a> was originally published in <a href="https://medium.com/bolt-labs">Bolt Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>