By Kathy Yang

Experimentation is an important part of product development at The New York Times — it helps us measure the impact of our changes, ship features safely, and gain insight into user behavior. Experiments are conducted by many different teams working across a diverse portfolio of products and projects, from Wordle to The Times’s Audio app to backend migrations. In the early days, this meant that teams would often develop their own approach to experimentation to address their specific requirements.

Over the past year, we have been on a journey toward standardization. There are many benefits to centralizing experimentation — having a single source of truth, observability into how all experiments impact important metrics, and easier comprehension of results from other teams (not to mention the time saved for individuals who would otherwise maintain their own pipelines!). On the flip side, teams had originally developed their own methodologies and processes for a reason — they were tailored for their unique needs.

At The New York Times, most product experiments use our internal experimentation platform, ABRA, which is short for A/B Reporting and Allocation architecture. Here’s a look at an early version. A lot has changed since those days, and we want to share some of the things we’ve learned on our standardization journey. While we use other types of experimentation, such as contextual multi-armed bandits, this article is focused specifically around the ways we’ve improved our A/B testing processes.

Experimentation requires good data and equally good metric governance.

As we all know, experiment results are only as trustworthy as the underlying metrics. A/B testing depends on the availability of high-quality metrics to analyze, and you cannot underestimate the importance of having a proper structure in place to manage these metrics.

We should anticipate that metric definitions are ever-evolving as our understanding of user behavior and business needs change. Thus, metric data and definitions should live outside of the experimentation platform. In ABRA, we have a clear delineation of responsibility for the data sources that feed ABRA’s Stats Engine (more on this in the next section): The platform team manages the exposure and assignment data and experiment configuration information, and analysts on other teams manage their metric data and definitions.

We set things up this way because it is crucial that the people who use these metrics on a day-to-day basis — for reporting, experimentation, modeling and other analytics — are the same ones who are responsible for creating, updating and deprecating them. Whether you are building or buying your Stats Engine, the individuals who use the experimentation metrics should be involved in their governance to help ensure they stay accurate and useful.

ABRA uses both enterprise metrics (think subscriber and revenue metrics), which are provided top-down from our central Enterprise Analytics team, as well as team metrics, which are provided by analysts on individual product teams. This dual-tiered structure ensures that enterprise metrics are defined consistently across all teams, while giving individual teams the freedom to add metrics specific to their domain.

When it comes to statistical tests, start with the basics.

While there are many more advanced techniques to explore, we found that we already began to feel the benefits of standardization once we had the basics down. This included conducting daily t-tests to estimate the difference in means between variants, and generating the point estimate, confidence interval, p-value, and the minimum detectable effect (MDE). At this point, teams that have their metrics hooked up to the platform have somewhere they can go to get automated results for their experiments, saving them from building their own.

Providing standardized reporting can be a catalyst for adopting best practices, but there is a significant cultural aspect to standardization as well. Consistent training, collaboration, and ongoing discussions with teams are essential to changing the way both technical and non-technical people think about experimentation. For example, if a team wasn’t consistently calculating confidence intervals or MDE before, the new results make it possible to quantify uncertainty in a way that was previously much more difficult. This needs to be paired with educating individuals on how to interpret these values, training analysts to conduct power analyses ahead of time, and changing stakeholders’ expectations to anticipate (and even insist on) seeing these values when presented with experiment results.

A lot has been written about ensuring data quality, including checking for sample ratio mismatch and capping skewed metrics. I want to expand on a couple more calculations that are particularly relevant at The Times:

Scaled impact

An increase of +1 page view per week per user can mean different things, depending on the population we’re experimenting on (e.g., all readers vs. international readers). Scaled impact is a useful way to present results when you want to compare experiments on different populations. It represents the estimated global increase we would have seen during the experiment if the treatment variant had been given to 100% of users, compared to if the control variant had been given to 100% of users.

To calculate scaled impact, you first start with the absolute difference in means between the treatment and control variants:

You then scale this up by the estimated number of users who were eligible for the experiment. This can be done by multiplying by the number of users exposed to the experiment n, and then dividing by the percent of users allocated to the experiment a (e.g., if you ran an experiment with a 20% control variant and a 20% treatment variant, a would be 40%):

Calculating impact on log-in rate

When a user is logged in, their activity is associated with their user ID. However, they can also have activity that is not tied to a user ID. For example, visiting the home page and playing Wordle can both be done without logging in or creating an account.

For these behaviors that can be done either while logged-out or logged-in, just looking at the impact on the metric itself may not tell the whole story. If a treatment variant increases (or decreases) users’ likelihood of logging in, they can end up with more (or less) of their activity associated with their user ID, even if their total activity didn’t change. When we see a statistically significant change in the log-in rate metric, this tells us that the results for these metrics may not be reliable, and we should not interpret them.

Provide paved paths for off-platform analyses.

Even with a robust experimentation platform, there will always be situations that require ad hoc analyses. In these cases, providing lightweight tooling or methodology guidelines can help teams move forward with a common approach. Here are some of the practices we use at The New York Times outside of ABRA:

Analyzing ad hoc metrics

Sometimes, the metrics that analysts need to examine are not in ABRA when they run the experiment. To make it easier to calculate the impact on these metrics after the fact, we have a Python notebook tool that approximates the calculations in ABRA’s Stats Engine. While Analysts need to provide their metrics manually, they can be confident that they’re using the agreed upon analysis methodology.

Quantile metrics

In addition to means, teams are often interested in quantile aggregations to measure Core Web Vitals (e.g., 90th percentile First Input Delay). To support quantile metrics, we have a separate pipeline that uses a bootstrapping technique to construct confidence intervals, which teams can use to check their experiments’ impact on Core Web Vitals. (Shoutout to The Times’s Ads Analytics team for building and scaling this resource!)

Non-inferiority testing

Not every experiment aims to improve metrics; sometimes, we just want to know that they aren’t degraded. In such cases, conducting a non-inferiority test, which tests whether the treatment is not worse than the control by at least some margin, can be more appropriate. Although ABRA was originally built for superiority testing, we have guidelines for adapting workflows to accommodate the non-inferiority approach. For example, even though the calculated p-value may not be relevant in this context, analysts can still rely on the confidence interval and assess whether the lower bound is greater than non-inferiority margin.

Eventually, some of these ad hoc tools might find their way onto the platform’s roadmap. Others might not. The key is to stay close to the practitioners on the ground, so that you develop a deep understanding of the common problems they face. Whether you are building or buying an experimentation platform, this will help you provide your analysts with the tooling and guidelines they need.

Experimentation is a core part of how we drive innovation here at the Times. There is still a lot of exploration to be done to standardize experimentation on all fronts, but we have learned some useful tips along the way. We’ve found establishing proper metric governance, prioritizing a short list of methodology essentials, and addressing some analysis needs off-platform to be important ingredients to success. We hope this sparks some ideas for your own experimentation journey!

Thank you to The Times’s Visualization and Experimentation team for their work on ABRA, the Ads Analytics team for pioneering early standardization work, the Analysts on the Experimentation work stream for helping develop analysis guidelines, and Kentaro Kaji and Mariam Melikadze for reviewing this post.

Kathy Yang is the director of Experimentation in the Data and Insights Group at The New York Times, where she works on developing experimentation methodologies, tooling and culture. Outside of work, Kathy enjoys yoga, traveling and reading.

Milestones on our Journey to Standardize Experimentation at The New York Times was originally published in NYT Open on Medium, where people are continuing the conversation by highlighting and responding to this story.