A/B Testing has been established as one of the best, if not the only, way to make unbiased product decisions and is an important tool to understand if you will be assuming a product management role. But how does this technique work and why is it so important?

Correlation vs. Causation

The first topic we need to tackle is correlation vs. causation. One of the most comical examples of this is pirates and global warming.

We can clearly see that as the number of pirates has decreased global temperatures have increased, but has global temperature increased because of a lack of pirates? Obviously not! Simply put, correlation is a measure of how related two variables are. The number of pirates in the world and global temperature are obviously correlated, but one does not have any influence on the other.

For a more real-world example, let’s assume that you redesign an advertisement for your website and notice a 5% increase in your conversion rate from this new ad. Just because your ad changed and your conversion rate went up 5% does not mean that your new ad caused this increase! In order to prove causation we need 3 things:

1. Correlation
2. Temporal Antecedence
3. No Extraneous Variables

In the above example we definitely have correlation, we have temporal antecedence (the ad redesign came before the 5% conversion rate increase), but we are unsure if there are any extraneous variables. Without controlling for all other factors we cannot be sure that our redesign is the only thing that contributed to the conversion rate increase. So how do we fix this? That’s where A/B testing comes in. The idea is to choose a subset of our total user base that is statistically similar to our total population of users and expose them to our new redesign (our treatment group), while the rest of our user base keeps seeing our original ad design (our control group). By doing this, the idea is that any extraneous variables will be constant across our control and treatment groups and are therefore eliminated in our analysis.

Sample Size Selection

The next logical question to ask ourselves is how many people do we need to test this on? Without going too deep into the weeds we will use the central limit theorem to make the assumption that if we take numerous samples from our population we have a normal distribution of means of those samples. This allows us to simplify the sample size calculation based on two factors, the power we wish to use and the significance level we want for our test.

Power can be thought of as correctly concluding that our treatment has an effect when it actually has an effect whereas significance level is the probability of concluding our treatment has an effect when it actually does not have an effect. Obviously we want a low significance level and a high power, but there is a trade-off between these two variables. Often people use a significance level of 5% and a power of 80%, but these numbers can and should be revised based on your needs. Once we have determined these values we must finally determine what the minimum detectable difference is between our test and control groups that we want to accept. In other words, what is the smallest amount of change, in our conversion rate for this example, that we will say is significant. After we have all of this we can plug into a fairly simple formula:

In this formula σ is the standard deviation of our sample, 1-β is the power, and α is the significance level. For a much more detailed description of power, significance, and the interplay between the two see this great post. The Z values can be obtained from a standard normal table or your favorite program. If we use a power of 80% and a significance of 5% they end up being about 0.845 and 1.96, respectively.

Run and Interpret Results in R

Okay, so we understand why A/B testing is important, and we now know how to determine how large of a sample size we need to run our test, but how do we actually run the test?

For this example we are going to assume we have an e-commerce site and we’re interested in how two different advertisements affect our average conversion rate. Let’s say we have data on 10,000 customers who have seen impressions for our site but only 20% of them actually click on those advertisements. Finally, we assume that we want to detect a minimum of 5% change in conversion rate. Luckily there is a function in R that lets us calculate the needed sample size given these proportions.

power.prop.test(p1 = .2, p2 = .25, power = 0.8, alternative = 'two.sided')

## Two-sample comparison of proportions power calculation
## n = 1093.739
## p1 = 0.2
## p2 = 0.25
## sig.level = 0.05
## power = 0.8
## alternative = two.sided
## NOTE: n is number in *each* group

So we will need 1,094 customers in our control and test groups. Great! Let’s generate some data.

First, generate two columns of data. The first will denote whether the user is part of the control or treatment group and the second will be the number of impressions this user has seen. For this A/B test we’ll assume that the only data we have is the number of impressions a user sees and whether they click on the ad or not. As long as we ensure that our treatment and control groups have the same average number of impressions we can safely assume that we have eliminated any extraneous variables. In reality you would want to randomize your control and treatment groups across devices, impressions, and other variables you would have access to in order to eliminate any doubts as to the effect of your new advertisement.

The code, available on my GitHub, will generate the data. Here we assume a Poisson distribution for impressions with a mean of 7 across our 10,000 users. We then randomly choose 1094 users for our control set and another 1094 for our treatment set. We perform a t-test to ensure that the control and treatment sets have the same mean number of impressions. This is a proxy for our elimination of extraneous variables.

t.test(impressions~treatment, data = AB.test.data)

## Welch Two Sample t-test

## data:  impressions by treatment
## t = 0.047934, df = 2185.1, p-value = 0.9618
## alternative hypothesis:true difference in means is not equal to 0
## 95 percent confidence interval:
## -0.2188938  0.2298628
## sample estimates:
## mean in group 0 mean in group 1
## 7.033821        7.028336

Looking at the density plot and the p-value of out t-test we can safely say that these two groups are similar in terms of impressions and this will be a valid A/B test.

The next step is to “run” the experiment. In reality you would observe the usage behavior of these two groups over an extended period of time and analyze the results. In this case we’ll just simulate the results with a conversion column. We simulate a 20% conversion rate for our control group and a 28% conversion rate for our treatment group (a little above our initial 5% minimum effect size). Now we could do a simple t-test to determine that the conversion rate is in fact higher for our treatment group, but often we will have more complex data. In that case we’ll want to understand the impact of the new advertisement. We can do a simple logistic regression and look at the output.

model.1 <- glm(conversion ~ as.factor(treatment), family = binomial(link = "logit"), data = AB.test.data)
summary(model.1)

We’ll look at only the coefficients of our model for a minute:

## Coefficients:
##                       Estimate  Std. Error z value Pr(>|z|)
## (Intercept)           -1.38515    0.07556 -18.332  < 2e-16 ***
## as.factor(treatment)1  0.44377    0.10118   4.386 1.15e-05 ***

We can see that our treatment variable is significant given the low p-value, and that the coefficient is positive, meaning that users exposed to our new ad were more likely to click on the ad.

This does not necessarily tell the whole story though. Another thing that is worth exploring is how impactful this new ad is compared to our old ad as a function of number of exposures. We’ll create another model where we look at the interaction of total exposures and our treatment group:

model.2 <- glm(conversion ~ impressions + impressions:as.factor(treatment), family = binomial(link = "logit"), data = AB.test.data)
summary(model.2)

Again, looking at the coefficients of this model:

## Coefficients:
##                                Estimate Std.Error zvalue Pr(>|z|)
## (Intercept)                      -4.29770 0.20000 -21.488 <2e-16 
## impressions                       0.37547 0.02436  15.413  <2e-16 
## impressions:as.factor(treatment)1 0.06912 0.01331   5.194 2.1e-07 

We see that all coefficients are again significant. Our main effect variable tells us that a higher number of impressions contributes to a higher likelihood of conversion, and our new interaction variable tells us that our new ad contributes to a higher likelihood of conversion among those who see more impressions. This is good news! With this model we can simulate the likelihood of customers clicking our ad with a given number of exposures, a much more powerful model than our original.

Conclusions

A/B Testing helps us to determine if changes we have made are contributing to the changes we may be seeing in our customer’s behaviors. It is an important topic to master as a product manager. We’ve run through a quick and simple example here, but the methodology can be applied to much more complicated A/B tests in the real-world. For the full R-code please visit my GitHub page and let me know if you have any questions or comments!

About the author

Ethan Nusbaum is a former Product Design Engineer and MBA passionate about utilizing data to make informed product decisions. He is expanding his data and product management knowledge daily and welcomes any advice or feedback.

On December 21, 2017 the Long Island Iced Tea Corp. rebranded itself as the Long Blockchain Corp. announcing that, while it would still be producing Long Island Iced Tea, it would also shift strategies and begin looking into potential partnerships, investments, and acquisitions in blockchain based companies. Their stock shot up as much as 500% in premarket trading.

On January 9, 2018 Kodak, who emerged from bankruptcy in 2013, announced that it would release KodakCoin, a new cryptocurrency “to empower photographers and agencies to take greater control in image rights management.” Their stock closed the next day over 200% higher.

It seems companies may have found a way to bolster their stock prices but are these gains warranted and sustainable? Can a beverage company successfully pivot into blockchain partnerships that will return value to shareholders? Will a camera company that is known for completely missing the digital transformation be able to successfully capitalize on blockchain technology, or does the term blockchain entice investors who are afraid they might be missing out on the next possible Bitcoin investment?

If you are thinking about whether you should be implementing blockchain technology into your business, it’s first important to have a decent understanding of what blockchain technology actually is. We can then start to see how this new technology might be used in different industries, and if it will truly change the world as many people claim, or if it merely has specific niche uses as others propose.

So What Is Blockchain?

There are tons of articles available online that will go through all of the technical details on how different blockchains work and delve into the cryptography being used on these different blockchains. Let’s just start with a very high-level explanation though. One of the simplest explanations I’ve heard for blockchain is to imagine an auditorium filled with people. Every person has a ledger where they record every transaction that takes place for a specified asset. Lets for a second assume that the asset here is Bitcoin. Say I want to transfer 5 Bitcoin to John Smith. I would get up on the stage and announce that Ethan Nusbaum is transferring 5 Bitcoin to John Smith. Everyone in the auditorium would check their individual ledgers to confirm that I have 5 Bitcoin to my name, and once that was agreed upon, everyone would record the transaction, reducing my account by 5 Bitcoin and increasing John Smith’s account by 5 Bitcoin.

This example beautifully conveys a few of the things about blockchain that people really like. You will often hear people claim that blockchain is decentralized, persistent, auditable, and anonymous, but what does that really mean?

In our auditorium we have a decentralized database of transactions. All of the ledgers contain a copy of every transaction, so no individual ledger could be stolen and manipulated. In other words, hacking our Bitcoin account records would be nearly impossible. This is often also referred to as a “trustless” system, because we do not need to trust any central authority in this transaction or even the person we are completing the transaction with. The persistence of blockchain technology refers to the fact that past transactions cannot be changed. Blockchain is append only — only new transactions can be added to our ledgers. Also, since every ledger has a copy of every transaction in history, we have a blockchain that is easily auditable. Finally, while in our example I stood on a stage and announced I was transferring Bitcoin to John Smith, in reality all accounts are anonymous, so while we can confirm that one account has enough to transfer to another, we do not actually know who owns which accounts. For a much more in-depth and technical discussion of how all of this works please watch this excellent video.

In short, blockchain removes many of the frictions in the transactional space by eliminating the need for a central, trusted authority, and having complete auditability and persistent records. We should be able to start to see how this could be very advantageous in many industries.

Where Can Blockchain Be Used?

As we’ve seen, people are trying to use blockchain almost everywhere. There are a few sectors however where we have seen more obvious advantages and I believe this technology will be extremely disruptive.

Finance

We don’t need to look far to see blockchain technology already being used by big banks. Many including J.P. Morgan with Quorum and Barclays and UBS are exploring how to incorporate blockchain technology into their services. Perhaps my favorite application right now though is Ripple. Anyone that has ever tried to wire money internationally already knows how burdensome and expensive this process is. Wiring money often takes 3–5 business days and there are high fees associated with these transactions. By reducing frictions in this process, Ripple claims they can cut fees by 60% and wire money almost instantaneously (source). The implications for people here could be huge, especially for migrant workers trying to send money back to their families but continually paying these high fees. This is truly a democratizing technology.

Identity

Blockchain technology could be used to help keep a verified, persistent proof of your identity. A startup in Austin I recently had the opportunity to speak with, SeriesX, is working on just this. By having this blockchain based employment and education history, you will be able to apply to jobs quicker and more easily while employers will be able to reduce their hiring costs and have a certified record of candidates past experiences. Possibly an even more revolutionary use of this technology could be in the voting sector. Within the US we seem to face a problem with voter registration every election. Voter suppression is a constant debate as law makers try to juggle voter fraud and voter suppression. With a blockchain verification of identity, however, there would be no way for voter fraud to persist. With the auditability of blockchain we could be sure that only people allowed to vote actually voted and with the distributed ledger platform there would be no way for votes to be miscounted. Follow My Vote is currently working on a solution here and it will be interesting to see what happens. Just this week, the Canadian government announced that it would run a trial using blockchain technology to enhance transparency in government funding. More transparency in politics is, in my opinion, always a good idea.

Real Estate

I recently had the opportunity to speak with Abhi Dobhal, VP of Business Development at Factom. I was fascinated to hear about some of the projects that Factom had worked on in the past including one with the government of Honduras. In Honduras, the records of land titles are not very sophisticatedly maintained and there are often disputes over who owns which parcels of land. The idea was to create a blockchain to maintain these land title records and eliminate any susceptibility to fraud. While the project ultimately fell through, other companies including Ubitquity are currently working on similar solutions. The idea is that these services once again reduce transaction costs, saving customers money, while also creating trusted and verified records of land titles.

Supply Chain and Logistics

Traceability is a problem that all logistics companies deal with. Season 2 of The Wire shows a Baltimore port where containers often go missing. While this is a TV show and obviously embellished, the logistics industry is plagued with missing packages and bureaucratic paperwork. How do they cut down on “lost” packages, find that missing customs form, or trace back the origin of tainted food? Blockchain technology can help with that. The auditability and persistence of the technology can help find the origin and stopping point of every shipment, and with a decentralized ledger we eliminate the possibility of a central system crash or tampering. This is exactly what Maersk and IBM are working on.

So What’s Next?

I hope now you have a better understanding of what Blockchain technology is and some of the potential use cases outside of just cryptocurrencies and finance.

I truly believe that it is a revolutionary technology that will drastically change multiple industries, but I don’t believe that it is for everyone.

If decentralization, auditability, and persistence can truly help you decrease your transaction costs and provide a better customer experience, then blockchain technology is definitely something you should explore. If you’re simply trying to capitalize on the buzz around the technology and are struggling to see how this could enhance your product and experience, I strongly believe you should not be trying to shoehorn a blockchain in.

The next few years in this space will be extremely interesting and I believe we will see a lot of innovation, but also a lot of poor use cases as people try to ride an emerging wave that they don’t quite understand or need. It’s the internet 2.0, strap in and enjoy the ride!

About the author

Ethan Nusbaum is a former Product Design Engineer and current MBA student at the University of Texas at Austin. He is actively looking to get involved with projects within this space.