Conditional Independence Made Simple
Conditional independence is a crucial building block of probabilistic graphical models (e.g., Bayesian networks). Having a solid understanding of this concept is a must before diving into Bayesian networks.
This post provides an intuitive introduction to the concept and then uses examples to make it crystal clear.
At a glance
- Independence: Two variables don’t affect each other.
- Dependence: Knowing one changes what we believe about the other
- Conditional independence: Two variables appear related, but once a third variable is known, their relationship disappears.
Dependence and Independence
Two random variables A and B are independent if one does not influence the other. Put simply: knowing one does not change what we believe about the other.
Let’s use a student example:
- A student’s intelligence influences their grade.
- A higher grade increases their chance of getting a recommendation letter.
This will be our running example throughout.
Now, for contrast, imagine linking student intelligence to the number of bicycles in the city 🚲. Clearly unrelated — knowing one tells us nothing about the other. That’s independence.
Two random variables A and B are independent if
P(A,B)=P(A) P(B)
Since from Bayes’ theorem we know that
P(A,B)=P(A) P(B∣A)
P(A,B) = P(A) P(B|A),
independence holds only if
P(B|A)=P(B) and P(A|B) = P(A)
Example: Independent distribution
Here’s a simple joint probability distribution of two independent binary variables with P(A=1)=0.5 and P(B=1)=0.5.
Each entry in the table satisfies the condition of P(A,B)=P(A)P(B). For example, P(A=1,B=1) = .5 * .5 = .25
Get Pankaj Chejara, PhD’s stories in your inbox
Join Medium for free to get updates from this writer.
| A | B | P(A,B) |
| — -| — -| — — — — |
| 0 | 0 | 0.25 |
| 0 | 1 | 0.25 |
| 1 | 0 | 0.25 |
| 1 | 1 | 0.25 |
Conditional Independence
Now let’s return to our student example:
- A highly intelligent student is likely to get a high grade.
- A high grade increases the chances of receiving a recommendation letter.
This gives us a Bayesian network:
Suppose we know that a student is highly intelligent. That increases the chances of a high grade, which in turn raises the probability of getting a recommendation letter.
Computation: Effect of intelligence on letter
We will now compute P(L=1|I=1). For simplicity sake, I have provided the table of needed probabilities instead of writing mathematical notations.
| I (Intelligence) | P(G=1|I) | P(L=1|G) | Contribution |
| — — — — — — — — — | — — — — — | — — — — — | — — — — — — — |
| 1 (High) | 0.7 | 0.8 | 0.56 |
| 1 (High) | 0.3 | 0.1 | 0.03 |
Adding these contributions:
P(L=1∣I=1)=0.59P(L=1|I=1) = 0.59
For a low-intelligence student, the probability is lower:
P(L=1∣I=0)=0.24P(L=1|I=0) = 0.24
These example illustrates how influence travels from intelligence to grade and then grade to letter. It clearly shows that Intelligence and Letter are dependent. In other terms, if we have information only about a student’s intelligence, then it updates our belief about the chances of that student getting a Letter.
👉 Intelligence matters when predicting whether a student will get a recommendation letter.
When grade is known
But what if we already know the grade?
- If G=1(high grade), the probability of a Letter is high — regardless of intelligence.
- If G=0(low grade), the probability of a Letter is low — again regardless of intelligence.
That’s conditional independence: Once we know the grade, intelligence adds nothing new in predicting the Letter.
✨ Key takeaway: Intelligence and recommendation letter only look dependent until grade is revealed. Once the grade is known, the letter ignores intelligence.
Summary: Independence means no influence. Dependence means knowledge updates our beliefs. Conditional independence means the relationship disappears once we know a “mediating” variable. In Bayesian networks, this is the key property that makes them powerful and efficient.
I hope you find this post helpful. In case if you have any suggestions or feedback, I would love to hear them.
