Looking at Test Results and Coin Flips the Bayesian Way
As you decided reading this post despite the uninviting title, let me try to guess why you might be here:
- you once knew Bayes, have good taste, and suspect a Bayes article might be fun (sometimes it is)
- you are prepping for an interview (consultancy role?) and heard rumors that the hiring manager likes Bayes questions
- while staring at some test result; an inner voice suggested a coin flip; a search landed you here, and you decided to ignore the weird ‘Bayesian’ bit for now(this one is a long shot)
- you are author’s girlfriend, sister or buddy (this, my friends, is a prime example of overfitting and data leaking at once)
Was I close? Please do leave a comment, especially if I’m way off!
Bayesian updating is a powerful idea. I’d heard about it before, but it truly clicked while reading David Spiegelhalter’s The Art of Statistics (highly recommend). The book is full of excellent real-world examples and helpful visual explanations, however, aiming general audience, it mostly avoids formulas.
For me, once I understand something, a compact formula describing the thing I understod is easier to remember later. Not having associated the examples in the book with formulas, I found myself knowing that interpreting the coin flips and test results (e.g., health scans) are essentially the same thing from the bayesian perspective, but not being able to solve them, without having to construct mental or physical frequency trees (see below). Searching the internet can be confusing: there are excellent explainers (such as the GeostatsGuy on YouTube, see the suggested references at the end of the post), but also posts with a heavy focus on the math, fancy terms and notation adapted specifically to the example that will follow. This post aims to bridge that gap by keeping one clean formula upfront, then looking at two seemingly unrelated cases.
If you are new to the topic, you may need to reactivate a few dormant neurons, but don’t give up. Once you understand the real-world implications, you’ll start looking forward to the next chance to reach for the Bayesian lens!
Bayes updating: a refresher
There are gentler intros (e.g. see the suggestions at the end), but here is an ultra compact refresher. Bayesian updating (or inference) refers to the process of updating (i.e., calculating a posterior of) our beliefs (i.e., prior), in the light of tests (i.e., evidence) and the likelihood (will expand on this below). The famous Bayes equation describes this relationship as follows:
In a more explicit form:
These terms can be understood as follows:
- P(A) — prior: probability of event A (sometimes called the ‘model’) happening (independent of B)
- P(B) — evidence: probability of event B (sometimes called the ‘data’) (independent of A)
- P(B|A) — likelihood: the probability of B given A, i.e., what would the data look like, for the cases our model (belief) holds
- P(A|B) — posterior: the probability A given B, i.e., in the light of data, how should we update our beliefs (model)
Now let’s take a look at two seemingly unrelated questions and see how Bayes can help us.
Case 1: Testing for a Rare Condition
‘Consider a certain condition with an occurrence rate of 1%. Our clinician is suspecting that we may have the condition and we take a test with 90% accuracy, meaning that 90% of the individuals having the condition will have a positive test result (i.e., sensitivity), and 90% of the individuals not having the condition will have a negative test result (i.e., specificity). And the bad news: test results are positive. Question is, how likely are we to have to actually have the condition?’ (Adapted from Spiegelhalter, 2019)
Let’s adapt the terms of the second form of the Bayes equation to our case to facilitate interpretation. Mapping event A to ‘having the condition’, abbreviated ‘H’, and event B to ‘positive test result’, abbreviated ‘+’, we can write the equation above as:
- P(H|+) : the probability of having the condition given a positive test.
- P(+|H) : the probability of getting a positive test result for an individual with the condition
- P(H) : probability of an inividual having the condition (in the absence of a test)
- P(+) : probability of getting a positive test result (for any individual)
The P(+), i.e., getting a positive test result is possible through i) a true positive, a positive result in the case of the event happening (i.e. having the condition), and ii) a false positive, a positive result in the case of the event not happenning (technically, the complement of P(H), which we will denote as P(N) for convenience). Thus, the above equation expands to:
Plugging in the numbers (P(H)=0.01; P(N)=0.99; P(+|H)=0.9; P(+|N)=0.1):
We get P(H|+) = 0.083, which is roughly 8%. If this sounds too low, think about the numbers in the denominator: only about (0.9*0.01/0.1*0.99*100) 9% of the time, a positive test result is a true positive. If you are still not convinced, take a look at this frequency tree:
What if the test was more accurate? Say, 95%? Plugging in the numbers (P(+|H) = 0.95, P(+|N) = 0.05), the probability rises to 16%, i.e., still not too high.
Get Onur Kerimoglu’s stories in your inbox
Join Medium for free to get updates from this writer.
Takeaway: for rare conditions, a positive test does not need to cause panic. Figure out the rarity of the condition and the test accuracy (better, sensitivity and specificity), then do the math, and hopefully find out that you are more likely to be a false positive (i.e., don’t have the condition) than a true positive (i.e., actually have the condition) — but don’t be that smartass arguing with your clinician, remember that the tests are one piece of evidence only, and not the whole story.
Case 2: Coin Flips
‘Consider three coins: one with two heads, one with two tails, and a fair one. You flipped a coin and it comes up heads. How likely is the other side coin to be also heads?’
Tempting answer:1/2, bayesian: not so fast.
Firstly, observe that the question can be translated into ‘what is the probability having chosen the coin with two heads?’. Defining:
- P(Hcoin): Probability of having chosen the heads coin (independent of the toss)
- P(Htoss): Probability of a heads toss (independent of the coin)
- P(Htoss|Hcoin): Probability of getting a heads toss, having chosen a heads coin
- P(Hcoin|Htoss): Probability of having chosen the heads coin, if a toss with it came up heads
We can rewrite the Bayesian updating formula as:
Now, let’s think about the probabilities:
- P(Htoss|Hcoin): obviously 1: with a heads coin, you can only get a heads toss
- P(Hcoin): 1/3 by definition, there are 3 coins and one of them is the heads
- P(Htoss): a bit less obviously 1/2: probability of having chosen the heads coin (1/3) plus probability of having chosen the fair coin multipled by 1/2, i.e., 1/3*1 + 1/3*1/2
Plugging the numbers in, we get 2/3:
Again, a frequency tree may help:
Plot twist: perhaps this question was too obvious. Let’s look at a slightly more difficult problem: we have the same three coins, and a flip with the chosen coin comes up heads. Now the question is: ‘how likely a second flip with the same coin will come up heads?’
One way of solving this is to simply make use of the results of the original question. We had calculated that the chances of having picked the heads coin to be 2/3. So we have a 2/3 probability that the next toss will be heads, if we picked the heads coin. In addition to this, we need to consider also the probability of having chosen the fair coin, probability which is 1–2/3 = 1/3, in which case there is a 1/2 chance of next toss coming up heads. So the additional probability of the second flip coming up heads through the fair-coin is 1/3*1/2 = 1/6. Summing up, the probability of the second flip coming up heads is (2/3 + 1/6) = 5/6
We can also solve it directly with Bayesian updating. Adapting the terms in our equation:
where,
- P(H): probability of heads coming up on the first toss: This is the sum of probabilities through the heads coin (1/3*1) and the fair coin (1/3*1/2), i.e., 1/3 + 1/6 = 3/6 = 1/2
- P(HH): probability of heads coming up two times with a chosen coin. This is the sum of probabilities through the heads coin (1/3*1*1) and the fair coin (1/3*1/2*1/2 = 1/12), i.e., 1/3 + 1/12 = 5/12
- P(H|HH): probability of the first toss coming up Heads given we had two heads (as dumb as it sounds, this is simply 1)
Plugging in, we get 5/6:
We can also see this result in the frequency tree:
Takeaway: think twice -whenever you have the luxury to do so!
I hope this was helpful and fun to read. Any thoughts? Anything you may want me to further expand on? Please leave comments.
For those who would like to take a deeper dive:
- David Spiegelhalter (2019). Art of Statistics: Learning from Data
- Michael J. Pyrcz (aka the GeostatsGuy on Youtube): interactive notebooks and run-through videos, e.g.: https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/Interactive_Bayesian_Updating.ipynb
And for fellow bloggers who might be wondering:
- Equation pngs: https://latex2png.com/
- Frequency trees: https://mermaid.js.org/
