Bayesian Learning
In the previous post, we used a Bayes Classifier for prediction.
Bayes Classifier (Naive Bayes style)
- Goal: Predict the class label Y for new data X.
- Focus: How likely is the class given observed features?
- Used in: Classification models like Naive Bayes.
In Bayesian Learning (Parameter Estimation)
- Goal: Learn/update the distribution of parameters w after seeing the entire training data i.e. input (X) and output (Y)
- Focus: How the model’s parameters should change based on data evidence.
- Used in: Bayesian inference / Bayesian linear regression.
Bayesian Model Selection is an approach to choosing the best model among several candidates using Bayesian probability theory. Unlike the frequentist view, which treats model parameters as fixed but unknown, the Bayesian approach treats parameters as random variables with prior distributions.
Two Approaches to Machine Learning
- Frequentist View
- Bayesian View
Frequentist View
- Core idea: Parameters are constant but unknown numbers
- Example: A coin factory produces identical coins
- Each coin has the same probability θ of getting heads
- θ is a single fixed value (e.g., 0.6)
- We don’t know it just yet - Learning process: Use maximum likelihood to estimate the parameter
- If we flip 10 coins and get 6 heads, 4 tails → θ = 6/10 = 0.6
Classical linear regression is Frequentist, as we find fixed parameters (weights) that minimize MSE or maximize likelihood.
Gradient Descent / Analytical Solution are both methods for estimating fixed parameters under the Frequentist assumption.
Logistic Regression / Cross-Entropy are Frequentist discriminative models as the parameters are estimated by maximizing the likelihood of the observed labels.
Probabilistic Linear Regression is still Frequentist unless we start assigning distributions (priors) to parameters, and that’s when we move to the Bayesian View.
Naive Bayes, LDA, and QDA are Generative models that can be viewed from a Frequentist or Bayesian lens. The Bayesian version explicitly assigns priors to model parameters.
Bayes' Theorem / Bayes Classifier introduces the Bayesian perspective, where probabilities represent degrees of belief, not just frequencies.
Bayesian View
Core idea: Parameters are random variables that satisfy a probability distribution
Example: A coin factory produces varying coins
- Different coins have different probabilities of heads
- θ itself follows a distribution (e.g., from 0 to 1)
- More realistic — even the best factory has variations!
Learning process: Update beliefs using prior + data
- Prior: Your belief before seeing data
- Posterior: Your updated belief after seeing data
- Formula: Prior + Data → Posterior
Example 1: Coin Flipping
Setup: Flip a coin 10 times → 6 heads, 4 tails
Frequentist View
- θ (probability of heads) is a single fixed number that exists but is unknown
- Analogy: “A perfect factory produces identical coins with the exact same θ.”
- Solution: Use maximum likelihood → θ = 6/10 = 0.6
- Just one number!
Bayesian View
- θ is NOT a fixed number — it follows a probability distribution
- Analogy: “Even the best factory has variation. Most coins might be 50% heads, but you’ll get 50.1%, 49.9%, etc.”
- θ could be anywhere from 0 to 1, following some distribution (e.g., uniform or peaked around 0.5)
- A whole distribution, not just one number!
Even with the tightest control, we still have variations. Variations mean we satisfy a probability distribution.
Example 2: Alien Heights
Setup: Measure heights of aliens: X₁, X₂, X₃, …
Frequentist View
- Heights follow Normal(μ, σ²) where μ and σ² are fixed values
- “The god of this planet used precisely these parameters.”
- We estimate them using maximum likelihood
Bayesian View
- μ is not fixed — μ follows a distribution
- σ² is not fixed — σ² follows a distribution
- Both parameters are themselves random variables with their own uncertainty
- Instead of assuming there is one single, true bell curve (the frequentist view), the Bayesian view considers a whole family of possible bell curves all at the same time.
- In the Bayesian view, the result isn’t a single bell curve. It’s a new, updated family of bell curves (a posterior distribution) that is now sharply “focused” around the curves that best explain the data we saw.
Why Bayesian is Better:
- Accounts for uncertainty in parameters
- Naturally prevents overfitting. HOW?
- Provides full distributions, not just point estimates
- A more realistic model of how the world works
The Trade-off of Bayesian:
- Much more computationally intensive
- Requires careful choice of prior
- Computationally complex, as updating distributions is mathematically difficult
But as the professor said: “Bayesian view is much more accurate, probably more realistic.”
Example
Coin flip: 6 heads, 4 tails out of 10 flips
- Frequentist: θ = 0.6 (done!)
- Bayesian: Start with a prior distribution for θ, update it with data, and get a posterior distribution for θ
Bottom line: Frequentist treats parameters as fixed unknowns. Bayesians treat parameters as distributions that get updated with evidence.
The Big Picture: How Bayesian Learning Works
The primary objective is to determine the probability distribution of the model’s parameters (e.g., the weights in a logistic regression) given the entire training set (X and Y).
Core Process: Prior Belief → See Data → Updated Belief (Posterior)
Now we’ll learn HOW to do this, updating mathematically.
Three Key Concepts in Bayesian Learning
There are three fundamental concepts you need to understand:
- Posterior Distribution — What we believe about parameters AFTER seeing data
- Posterior Predictive Distribution — How we make predictions for NEW data
- Marginal Likelihood — How well our model explains the observed data
Let’s break down each one…
Concept 1: Posterior Distribution
“Summary of knowledge AFTER seeing the data”
p(w|X, Y)
where,
- w = parameters (like θ in the coin example, or μ and σ in the alien height example)
- X = input data (features)
- Y = output data (labels/observations)
- p(w | X, Y) = “What is the probability distribution of parameters w, given that we’ve seen data X and Y?”
Since we’re interested in finding w, and p(Y|X) doesn’t depend on w, we can write: p(w∣X,Y) ∝ p(Y∣X,w)⋅p(w)
This reads as: “Posterior is proportional to Likelihood times Prior”
The denominator p(Y|X) is constant with respect to w, so we can ignore it when comparing different values of w.
Example: Back to Our Coin
Prior: We believe θ (probability of heads) follows some distribution
- Maybe we think most coins are fair, so θ is probably around 0.5
Likelihood: We flip coins and get 6 heads, 4 tails
- This data tells us something about θ
Posterior: Combine prior belief + observed data
- Our updated belief about θ after seeing 6 heads and 4 tails
- This is still a distribution, not just one number!
Key Insight from Professor:
“Before you see the data, you have a prior. After seeing the data, your θ still satisfies a probability distribution (the posterior), not a fixed number.”
Concept 2: Posterior Predictive Distribution
“Given everything I’ve learned from past data (X, Y), what do I predict for a NEW input x?”*
What’s happening here?
- x*: This is a new, unseen input data point. It’s the new piece of information we want to predict about.
- y*: This is the expected output (or label) for that new input x*.
p(y* | x*, w): If the parameter was w, what would we predict for the new input x*?p(w | X,Y): How likely is each value of w? (This is our posterior!)Integration ∫: Average predictions over ALL possible values of w, weighted by their probability
Why Integration?
Remember: In the Bayesian view, w is not a single number. It’s a distribution!
- We don’t know exactly which w is correct
- So we consider ALL possible values of w
- We weight each prediction by how likely that w is (according to the posterior)
- This gives us a distribution of predictions, not just one prediction
Concept 3: Marginal Likelihood (Model Evidence) / Likelihood
“Before seeing the data, how likely is this data according to my model?”
What’s happening here?
p(Y|X, w): If the parameter was w, how likely is this data?p(w): How likely is each value of w? (This is our prior!)Integration ∫: Average over ALL possible values of w according to our prior belief
Key Point: Uses PRIOR, Not Posterior!
“Why use the prior distribution for Marginal Likelihood?”
Two important reasons:
- Evaluates the model’s predictive ability BEFORE seeing data
- This is a fair test: “Could my model have predicted this data?”
- Uses only prior beliefs, not cheating by looking at the answer - Prevents overfitting
- Using prior encourages simpler models
- Complex models are only favored if there’s strong evidence in the data
- Natural “Occam’s Razor” built into the math!
How We Use Marginal Likelihood
Model Selection: Compare different models by their marginal likelihood
- Model A: p(Y|X) = 0.001
- Model B: p(Y|X) = 0.15
- Choose Model B! (higher marginal likelihood)
“Choose model with highest p(Y|X)”
The model that makes the observed data most probable (according to prior beliefs) is the best model.
Putting It All Together: The Bayesian Workflow
Step-by-Step Process
1. Start with Prior
- p(w): Your belief about parameters before seeing any data
- Example: “I think coins are probably fair, so θ is around 0.5”
2. Observe Data
- Collect data (X, Y)
- Example: Flip coins, get 6 heads and 4 tails
3. Calculate Posterior
- p(w|X,Y) = p(Y|X,w) · p(w) / p(Y|X)
- Update your belief about w using Bayes’ rule
- Example: “Okay, θ is probably a bit higher than 0.5, maybe around 0.55–0.65”
4. Make Predictions (if needed)
- p(y* | x*, X, Y) = ∫ p(y* | x*,w) · p(w | X,Y) dw
- Predict for new inputs by averaging over the posterior
- Example: “Next flip will probably be heads with 60% confidence”
5. Evaluate Models (if comparing)
- p(Y | X) = ∫ p(Y|X,w) · p(w) dw
- Compare which model best predicted the data
- Example: “My coin model is better than assuming all flips are independent”
Professor’s warning:
“Even for simple cases, updating from prior to posterior is very hard. The mathematics is complex.”
Summary: The Three Key Formulas
Bayesian Learning in One Sentence: Start with prior beliefs about parameters, update those beliefs when you see data (get posterior), then use the full distribution of parameter beliefs to make predictions and compare models.
Beta-Bernoulli Model
The Beta-Bernoulli model is a classic Bayesian example for coin flipping (or any binary outcome).
The Setup:
- We have a coin with an unknown probability θ of landing heads
- We flip it n times
- We want to learn about θ using a Bayesian approach
Why This Model?:
- Simple enough to understand completely
- Shows ALL three Bayesian concepts (posterior, predictive, marginal likelihood)
- Has a beautiful mathematical property called conjugacy
The Three Components of the Model
1. The Data Model (Likelihood): Bernoulli Distribution
Single coin flip: p(x | θ) = θˣ (1−θ)¹⁻ˣ
where:
→ x = 1 if heads, x = 0 if tails
→ θ = probability of heads
Multiple coin flips (n flips): p(x₁,…,xₙ | θ)= θⁿ¹ (1−θ)ⁿ⁰
where:
→ n₁ = number of heads (ones)
→ n₀ = number of tails (zeros)
→ n = n₁ + n₀ (total flips)
Example: Flip 10 times, get 6 heads and 4 tails
- n₁ = 6, n₀ = 4
- p(data|θ) = θ⁶ (1-θ)⁴
2. The Prior: Beta Distribution
where:
→ θ is between 0 and 1 (it’s a probability!)
→ α, β are hyperparameters (we choose these)
→ B(α, β) is a normalizing constant (makes probabilities sum to 1)
Understanding the Beta Distribution
Think of Beta(α, β) as encoding pseudo-observations from our “prior life”:
“Beta distribution can be thought of as a bias with pseudo-counts α-1 successes and β-1 failures.”
What this means:
→ α − 1= imaginary heads we’ve seen before
→ β − 1 = imaginary tails we’ve seen before
→ These represent our prior belief
3. The Posterior: Still Beta! (Conjugacy)
This is Beta(α+n₁, β+n₀)!
The Magic of Conjugacy
Conjugate Prior: When prior and posterior belong to the same family of distributions
“Beta distribution is a conjugate prior to Bernoulli likelihood.”
What this means:
- Prior: Beta distribution
- Likelihood: Bernoulli/Binomial
- Posterior: Also Beta distribution!
Why this is amazing:
- Makes calculations much easier
- Just update the parameters: α → α+n₁, β → β+n₀
- Don’t need complex integrals!
The Simple Update Rule:
Prior: Beta(α, β)
See data: n₁ heads, n₀ tails
Posterior: Beta(α+n₁, β+n₀)Intuitive meaning:
- Start with α-1 imaginary heads, β-1 imaginary tails
- Observe n₁ real heads, n₀ real tails
- End with (α-1 + n₁) total heads, (β-1 + n₀) total tails
Example Walkthrough
Let’s work through a complete example:
Setup
- Prior: Beta(2, 2)
- Means: α-1 = 1 prior head, β-1 = 1 prior tail
- Weak belief that the coin is fair - Data: Flip 10 times, observe 6 heads, 4 tails
- n₁ = 6, n₀ = 4
Step 1: Calculate Posterior
Using the update rule:
p(θ∣X) = Beta(α+n1, β+n0)
p(θ∣X) = Beta(2 + 6, 2 + 4)
p(θ∣X) = Beta(8, 6)Interpretation:
- Started with 1 imaginary head, 1 imaginary tail
- Saw 6 real heads, 4 real tails
- Now have 7 total heads, 5 total tails worth of evidence
- Belief shifted slightly toward θ > 0.5
Question:
What does the parameter θ (theta) represent in the context of the
Beta distribution for coin flips?
Answer:
θ represents the probability of seeing heads (H) in a coin flip.
It's the parameter we're trying to estimate, and it can take values
between 0 and 1.Question:
You believe a coin might be biased toward heads and set your prior
as Beta(3, 2). After 20 flips, you observe 15 heads.
a) What is your posterior?
b) What would the frequentist estimate be?
c) How do they compare?
Answer:
a) Posterior: Beta(3 + 15, 2 + 5) = Beta(18, 7)
Bayesian estimate (mean): 18/(18+7) = 18/25 = 0.72
b) Frequentist estimate: 15/20 = 0.75
c) The Bayesian estimate is slightly lower because the prior Beta(3,2)
had mean 3/5 = 0.6, which pulls the estimate down slightly
from the pure data estimateQuestion:
What would Beta(1, 1) represent as a prior? What about Beta(100, 100)?
Answer:
Beta(1, 1)
This is like starting with just one imaginary success α = 1 and one imaginary
failure β = 1.
Because we have minimal, perfectly balanced information, we have no reason
to prefer any probability over another.
The math for this results in a perfectly flat, uniform line from 0 to 1.
Every possible probability (0.2, 0.5, 0.8, etc.) is considered equally
likely at the start.
Beta(100, 100)
This is like starting with a lot of imaginary data: 100 successes α = 100 and
100 failures β =100.
This large, balanced dataset creates a very strong, confident prior belief
that the true probability is extremely close to 0.5.
The large total count (α + β = 200) makes this belief very "stiff" and
hard to change, resulting in a very sharp, narrow peak centered at 0.5
(low variance).Question: Posterior Calculation with Observed Data
Q: Suppose you have a coin and you want to estimate the probability of
getting heads (θ). Your prior belief is represented by Beta(2, 2).
You flip the coin 5 times and observe the following sequence: H, H, T, T, T
(a) What is the posterior distribution after observing this data?
(b) Calculate the posterior mean, which represents your updated estimate
of the probability of getting heads.
(c) Show all the calculations step by step.
Answer (a): Finding the Posterior Distribution:
Given:
Prior distribution: θ ~ Beta(α = 2, β = 2)
Observed data: H, H, T, T, T
Number of heads (x) = 2
Number of tails = 3
Total flips (n) = 5
Posterior Update Rule:
For Beta-Binomial conjugate pair:
Posterior: θ∣data ∼ Beta(α+x,β+(n−x))
Calculation:
α_posterior = α + (number of heads) = 2 + 2 = 4
β_posterior = β + (number of tails) = 2 + 3 = 5
Hence, Posterior distribution is Beta(4, 5)
Answer (b): Posterior Mean (Expected Probability of Heads)
The mean of a Beta(α, β) distribution is:
E[θ] = α / (α+β)
For our posterior Beta(4, 5):
E[θ∣data]= 4 / (4+5)
= 4/9
Hence, The probability of getting heads = 4/9 ≈ 0.444
Answer(c): Step-by-Step Summary:
1. Start with prior: Beta(2, 2)
Prior mean: 2 / (2+2) = 1 / 2 = 0.5
2. Observe data: 2 heads, 3 tails
3. Update parameters:
- New α = 2 + 2 = 4
- New β = 2 + 3 = 5
4. Posterior: Beta(4, 5)
Posterior mean: 4 / (4+5) ≈ 0.444
Interpretation:
We started believing the coin was fair (prior mean = 0.5)
After observing more tails than heads (3 tails vs 2 heads), our belief updated
We now estimate the probability of heads is 4/9, which is less than 0.5
This makes sense since we observed more tails in our sampleMaking Point Estimates: MAP
Sometimes we want a single value for θ (not an entire distribution).
Intuition:
- Numerator: total “successes” (prior + observed)
- Denominator: total “trials” (prior + observed)
- It’s like a weighted average!
Example Calculation:
With α=2, β=2, n₁=6, n₀=4:
Compare to the Maximum Likelihood Estimate (Frequentist):
Notice:
- MAP is slightly lower than MLE
- Prior (believing fair coin) pulls estimate toward 0.5
- As n increases, prior matters less
The Effect of Data vs Prior
“As n increases, the data overwhelms the prior. Thus the prior is similar to regularization.”
Small data (n=2, 2 heads):
- Prior: Beta(2,2) → α-1=1 prior head, β-1=1 prior tail
- MAP: (1+2)/(2+2–2+2) = 3/4 = 0.75
- MLE: 2/2 = 1.0
- Prior has a strong influence!
Large data (n=100, 60 heads):
- Prior: Beta(2,2)
- MAP: (1+60)/(2+2–2+100) = 61/102 ≈ 0.598
- MLE: 60/100 = 0.6
- Data dominates, prior almost irrelevant!
This is like regularization:
- Prior prevents overfitting with small data
- Prior becomes negligible with large data
- Built-in “smoothing” of estimates
Making Predictions: Posterior Predictive Distribution
Now the key question: “What’s the probability the next flip is heads?”
Three Different Approaches
Let’s use the example from slides: α=β=1, observed 2 heads (n₁=2, n₀=0)
Approach 1: Frequentist (MLE)
“I saw 2 heads out of 2 flips, so the probability for the next flip to be a head is 100%”
Problem: Overconfident! Only 2 flips!
Approach 2: MAP (Single θ)
With a uniform prior, MAP = MLE here
- Same problem: Still overconfident!
Approach 3: Bayesian (Average over all θ)
With α=1, β=1, n₁=2, n₀=0:
“75% chance of heads”
- Much more reasonable! Accounts for uncertainty
The Posterior Predictive Formula
Breakdown:
- Numerator: α + n₁ = prior heads + observed heads
- Denominator: α + β + n = prior total + observed total
Key insight:
“Bayesian averaging over possible models”
- We don’t commit to single θ
- We average predictions across ALL possible θ values
- Weight each θ by its posterior probability
- Naturally incorporates uncertainty!
Complete Example with All Three Methods
Why Posterior Predictive is best:
- Accounts for small sample size
- Prior (uniform) adds “imaginary” 1 head and 1 tail
- Natural uncertainty quantification
- Prevents overconfidence
Summary: The Complete Beta-Bernoulli Model
- Start with prior belief: Beta(α, β)
- Like having seen α-1 heads, β-1 tails before - Observe data: n₁ heads, n₀ tails
- Update to posterior: Beta(α+n₁, β+n₀)
- Just add observations to prior! - Make predictions:
- Point estimate (MAP): (α-1+n₁)/(α+β-2+n)
- Full Bayesian: (α+n₁)/(α+β+n)
Bayesian Linear Regression
Regular (Frequentist) Linear Regression:
- Find the single best w that fits the data
- Solution: w = (Xᵀ X)⁻¹ Xᵀ Y
- We get this equation from the Analytical Solution
Bayesian Linear Regression:
- Find a distribution over w
- Accounts for uncertainty in parameters
- Uses prior beliefs about w
“In frequentist view (Probabilistic Linear Regression), we just optimized the negative log likelihood and found the best parameter w. Here, we find distribution of w based on the assumed prior distribution of w.”
Problem Setup
Components:
- X ∈ ℝⁿˣᵈ: Design matrix (n observations, d features)
- Each row is one data point
- Each column is one feature - Y ∈ ℝⁿ: Response vector (n observations)
- w ∈ ℝᵈ: Model parameters (weights we want to learn)
- ε: Gaussian noise with variance σ²
Example: Predicting house prices
- n = 100 houses
- d = 3 features (size, bedrooms, age)
- X is a 100×3 matrix
- Y is a 100×1 vector of prices
- w is a 3×1 vector of weights
Get Prajun Trital’s stories in your inbox
Join Medium for free to get updates from this writer.
The Three Components (Same Pattern as Beta-Bernoulli!)
1. Prior: Gaussian Distribution on w
What this means:
- Before seeing ANY data, we believe w is centered around 0
- Each weight is independent (identity covariance matrix I)
- Standard deviation of 1 for each weight
Interpretation:
“This prior reflects our belief that w is centered around zero with unit variance.”
- We expect weights to be small
- No extreme values
- This is a regularization assumption!
Why this prior makes sense:
- In most real problems, weights shouldn’t be huge
- Protects against overfitting
- Similar to L2 regularization (we’ll see this connection!)
2. Likelihood: How Well Does w Explain the Data?
What this means:
- Given weights w, predictions are Xw
- Actual observations Y differ from predictions by Gaussian noise
- Noise has variance σ² (same for all observations)
- The likelihood captures the noise model in the data Y
The exponent term:
Frequentist connection:
“What we did in linear regression was we adapted frequentist view. We thus simply optimized the negative log likelihood and found the best parameters w.”
- Frequentist: Maximize likelihood → Minimize ||Y-Xw||²
- Bayesian: Use likelihood as part of Bayes' rule
3. Posterior: Updated Belief After Seeing Data
where:
- Likelihood: p(Y |X , w ) = N(Y |Xw , σ²I )
- Prior: p(w ) = N(w |0,I )
The Magic of Conjugacy:
“Since both prior and posterior are Gaussian and Gaussian is conjugate distribution of itself, the posterior is also Gaussian”
Posterior parameters:
We need to derive the posterior, p(w | X, Y) in closed form by “completing the square”. We know p(w∣X,Y) ∝ p(Y∣X,w) ⋅ p(w) that is, we need to multiply the likelihood and the prior.
Deriving the Posterior: “Completing the Square”
This is the mathematical magic that makes it work. Let’s break it down step by step.
Step 1: Write Out Likelihood and Prior in Exponential Form
Step 2: Multiply Likelihood and Prior
The posterior is proportional to the product of the likelihood and the prior:
Step 3: Collect Terms Involving w
First, gather all the terms involving w (these form the exponent):
Suppose:
- A = (Xᵀ X)/σ² + I
- b = (Y Xᵀ)/σ²
Step 4: Complete the Square
Standard result from linear algebra:
The last term (bᵀ A⁻¹ b) doesn’t depend on w, so we can get rid of that. Now, substituting back into the posterior expression, we get:
Step 5: Identify Mean and Covariance
Mean (μ):
Similarly, Covariance (∑):
After multiplying both the above equations by σ², the σ² in the denominator cancels out, and we get this mean and covariance. Hence, this is a Gaussian distribution with:
- Mean: w_post = A⁻¹ b = (XᵀX + σ²I)⁻¹ Xᵀ Y
- Covariance: Σ_post = A⁻¹= σ² (Xᵀ X + σ²I)⁻¹
Since both prior and posterior are Gaussian and Gaussian is a conjugate distribution of itself, the posterior is also Gaussian:
The posterior distribution is: p(w|X, Y) = N(w | w_post, Σ_post)
where,
- Mean: w_post = A⁻¹ b = (XᵀX + σ²I)⁻¹ Xᵀ Y = 1/σ² ΣXᵀ Y
- Covariance: Σ_post = A⁻¹= σ² (Xᵀ X + σ²I)⁻¹
Understanding the Posterior Parameters
The Mean: w_post = (XᵀX + σ²I)⁻¹ Xᵀ Y
Compare to Frequentist (OLS): w_OLS = (XᵀX)⁻¹ Xᵀ Y
We get this equation from the Analytical Solution.
Key difference: The σ²I term!
- Acts as a regularization
- Makes the matrix invertible even if XᵀX is singular
_ Shrinks weights toward zero
Special cases:
- No noise (σ → 0):
- w_post →(XᵀX)⁻¹ Xᵀ Y(pseudo-inverse solution)
- Covariance → 0 (no uncertainty)
- We’re completely certain about w - Lots of data (n → ∞):
-XᵀXgrows large
- σ²I becomes negligible
- w_post → w_OLS (Bayesian and Frequentist agree!)
The Covariance: Σ_post = σ² (Xᵀ X + σ²I)⁻¹Dimensions: d×d matrix
What it tells us:
- Diagonal elements: Uncertainty in each weight
- Off-diagonal: Correlations between weights
Key insight:
“As the number of samples increases, the covariance shrinks as XᵀX increases and thus its inverse has to decrease.”
- More data →
XᵀXgets bigger - Bigger
XᵀX→(XᵀX + σ²I)⁻¹gets smaller - Smaller covariance → less uncertainty!
This makes intuitive sense:
- Little data → very uncertain about w
- Lots of data → very certain about w
Maximum A Posteriori (MAP) Estimate
Sometimes we want a single point estimate instead of the full distribution.
Since posterior is Gaussian:
- Gaussian is maximized at its mean
- Therefore: w_MAP = w_post
Connection to Ridge Regression!
Ridge regression is secretly Bayesian learning! When we add regularization, we are actually using a Bayesian prior.
Ridge regression minimizes:
Or in simple terms:
Ridge regression minimizes:
where:
- First term: Fit the data (minimize prediction error)
- Second term: Keep weights small (regularization)
- λ: Regularization parameter
Bayesian Prior for Ridge Regression
In Bayesian learning, we assume: 𝑤 ~ 𝒩(0, 𝐼)
Meaning:
- Weights are drawn from a Gaussian distribution
- Mean = 0 (we want weights close to zero to keep the model simple)
- Covariance = I (identity matrix — all weights independent, same variance)
The Mathematical Connection
Step 1: Write the Prior Probability
Step 2: Apply Logarithm
Step 3: Combine with Likelihood
The full objective becomes:
σ² depending on variance)Theorem:
MAP (Maximum A Posteriori) estimate with Gaussian prior 𝒩(0, σ²I) is identical to ridge regression λ = σ²
Why This Is HUGE
“Adding regularization makes training stable and generalization better. It is implemented by adding a lambda to the diagonal of the Jacobian matrix. Which reduces the condition number and makes the Jacobian invertible too.”
“Why did this work? People didn’t understand before, but now we know: Because it is the same as using a prior. Thus, Bayesian learning.”
The insight:
- Ridge regression has been used for decades
- People knew it worked, but didn’t fully understand why
- Now we know: It’s equivalent to Bayesian inference with a Gaussian prior!
Practical benefits:
- Numerical stability: XᵀX + σ²I is always invertible
- Better generalization: Prior prevents overfitting
- Theoretical justification: It’s Bayesian inference!
“Thus, we are doing Bayesian learning when we use regularization.”
Every time we use ridge regression, we are implicitly doing Bayesian inference!
Extension: L1 Regularization (Lasso)
What if we use Lasso instead?
Bayesian interpretation: The prior is now a Laplacian distribution instead of a Gaussian.
So,
Ridge (L2) ↔ Gaussian prior
Lasso (L1) ↔ Laplacian prior
Making Predictions: Posterior Predictive Distribution
The posterior predictive distribution describes the distribution of a new observation y* given the data X and Y
Now we want to predict y for a new input x* (not in training data).
What’s happening:
- p(y*|x*, w): Prediction for each possible w
- p(w|X, Y): How likely is each w? (posterior)
- Integration: Average over all w, weighted by probability
Computing the Integral
We already know that:
→ p(y*|x*, w) = N(y*|x*ᵀ , w, σ²) (likelihood for new point)
→ p(w|X, Y) = N(w | w_post, Σ_post) (posterior)
Substituting the above known value into the Predictive Distribution.
The result of this integral is:
This is a Gaussian distribution with:
Mean: E[y* | x*, X , Y ] = x*ᵀw_post
Variance: Var[y* | x*, X , Y ] = σ² + x*ᵀ Σ_post x*
σ²: Observation noise (irreducible)
- Even with the perfect model, data has noise
x*ᵀ Σ_post x*: Model uncertainty
- Uncertainty in w propagated to the prediction
- Depends on how much data we have
Distribution of y is centered around the MAP estimate, and the variance is the sum of noise in observation (σ²) and sum of model uncertainty (variance and covariance) along different axis x*ᵀ Σ_post x*.”
Understanding the Prediction Variance
Two sources of uncertainty:
- Aleatoric uncertainty (σ²):
- Noise in the data
- Irreducible
- Same everywhere - Epistemic uncertainty (x*ᵀ Σ_post x*):
- Uncertainty in model parameters
- Reducible with more data!
- Varies depending on x*
Example:
- Training data: x from 0 to 10
- Predict at x* = 5: Low epistemic uncertainty (in the middle of the data)
- Predict at x* = 20: High epistemic uncertainty (extrapolating!)
Key insight:
- Bayesian approach quantifies both types of uncertainty
- Frequentist only accounts for observation noise
- This is why Bayesian is more honest about uncertainty!
Model Comparison: Marginal Likelihood (Evidence)
What’s happening:
- Integrate over ALL possible w according to prior
- Asks: “How likely is this data before seeing it?”
This quantity is crucial for:
- Model selection (choosing between different models)
- Hyperparameter optimization
Dimensions check:
- Y is n×1
- Mean is n×1 (all zeros)
- Covariance is n×n (σ²I + XX^T)
- X is n×d, so XXᵀ is n×n
Notice that XXᵀ is of dimension n×n as it should be for the covariance matrix of Y, which is n-dimensional.
How to Use Marginal Likelihood
Two main applications:
- Model Selection:
- Marginal likelihood provides a way to compare different models and select among them
- Compare models with different features
- Choose model with highest p(Y|X)
- Automatically trades off fit vs complexity! - Hyperparameter Optimization:
- Provides an approach to optimize the hyperparameter (σ²) by maximizing the marginal likelihood
- Try different values of σ²
- Pick σ² that maximizes p(Y|X)
- This is “empirical Bayes”
Example: Comparing polynomial degrees
- Model 1: Linear (d=1)
- Model 2: Quadratic (d=2)
- Model 3: Cubic (d=3)
- Compute p(Y|X) for each
- Choose the model with the highest marginal likelihood
The Bayesian Linear Regression Pipeline
Key Takeaways
- Conjugacy is beautiful: Gaussian prior + Gaussian likelihood = Gaussian posterior
- Ridge regression = Bayesian inference:
- Ridge with λ = σ² is equivalent to MAP with a Gaussian prior
- Regularization is just a prior belief! - Uncertainty quantification:
- Covariance shrinks with more data
- Predictions include both noise and model uncertainty - More data = Less uncertainty:
- XᵀX grows → Σₚₒₛₜ shrinks
- Prior matters less, data dominates - Marginal likelihood for model selection:
- Automatic Occam’s razor
- No need for cross-validation! - Everything connects:
- Prior → Posterior → Predictions → Model comparison
- One unified framework!
Bayesian Inference with Discrete Priors: Detailed Solution
Problem Setup
Dataset: {H, T, H, H, T}
- 3 heads (H)
- 2 tails (T)
Goal: Find the posterior distribution of θ (probability of getting heads)
Two different priors to compare:
- Prior 1: θ can be {0.1, 0.3, 0.5, 0.7, 0.9}
- Prior 2: θ can be {1/6, 1/2, 5/6}
We’ll use Bayes’ theorem to update our beliefs after seeing the data.
The posterior distribution is given by:
Components:
- P(θ|Data): Posterior — what we want to find
- P(Data|θ): Likelihood — how well θ explains the data
- P(θ): Prior — our initial belief about θ
- P(Data): Marginal likelihood — normalizing constant
The Likelihood Function
For our data (3 heads, 2 tails), the likelihood is: P(Data | θ) = θ³ (1−θ)²
Why this formula?
- θ³: Probability of getting 3 heads
- (1-θ)²: Probability of getting 2 tails
- This is a Binomial distribution
Solution for Prior 1
Step 1: Define the Prior
Prior 1: θ ∈ {1/10, 3/10, 5/10, 7/10, 9/10} = {0.1, 0.3, 0.5, 0.7, 0.9}
Each is equally likely, so P(θ) = 1/5
Step 2: Calculate Likelihood for Each θ
Use the formula: P(Data | θ) = θ³ (1−θ)²
Step 3: Calculate Marginal Likelihood (Normalizing Constant)
Since P(θ) = 1/5 for all θ, that is, we have 5 priors and we are taking the average of those.
What does this mean?
- This is the overall probability of seeing our data (3H, 2T) across all possible θ values
- It’s our “evidence” for this model
Step 4: Calculate Posterior for Each θ
Now, the posterior probabilities for each θ are:
Check: 0.0097 + 0.1585 + 0.3745 + 0.3699 + 0.0874 = 1.0
Interpretation (Prior 1)
Before seeing data:
- All θ values are equally likely ((1/5) that is 20% each)
After seeing 3 heads, 2 tails:
- θ = 0.5: 37.5% probability ← Most likely!
- θ = 0.7: 37.0% probability ← Almost as likely!
- θ = 0.3: 15.9% probability
- θ = 0.9: 8.7% probability
- θ = 0.1: 1.0% probability ← Almost ruled out
Key insight: Data shifted belief toward θ ≈ 0.5–0.7 (makes sense: 3/5 = 0.6)
Solution for Prior 2
Step 1: Define the Prior
Prior 2: θ ∈ {1/6, 1/2, 5/6} ≈ {0.167, 0.5, 0.833}
Each is equally likely, so P(θ) = 1/3
This prior is different:
- Only 3 possible coins (not 5)
- More extreme values (1/6 and 5/6 vs 0.1 and 0.9)
Step 2: Calculate Likelihood for Each θ
Use the formula: P(Data | θ) = θ³ (1−θ)²
Step 3: Calculate Marginal Likelihood
Since P(θ) = 1/3 for all θ:
That is 393/23328.
Step 4: Calculate Posterior for Each θ
Check: 0.0636 + 0.6183 + 0.3180 ≈ 1.0
Interpretation (Prior 2)
Before seeing data:
- All three coins are equally likely (33.3% each)
After seeing 3 heads, 2 tails:
- θ = 1/2: 61.8% probability ← Most likely by far!
- θ = 5/6: 31.8% probability ← Still plausible
- θ = 1/6: 6.4% probability ← Unlikely
Key insight: Fair coin dominates, but a biased-toward-heads coin still has a significant probability.
Model Selection: Comparing the Two Priors
We now have two different models (Prior 1 and Prior 2). Which one is better?
The Question
We select the model with the highest marginal likelihood.
Why marginal likelihood?
- It tells us: “How well did this model predict the data?”
- Higher marginal likelihood = better model
- Automatically balances fit and complexity!
So, Prior 2 is chosen.
Prior 2 is slightly better because:
- Higher marginal likelihood: It assigns a slightly higher probability to the observed data
- Reasonable coin values: The three coins in Prior 2 {1/6, 1/2, 5/6} cover the space reasonably well
- Not too complex: Only 3 coins vs 5 coins, so it’s simpler but still captures the data well
Prior 1 Evolution
BEFORE DATA (Prior):
θ: 0.1 0.3 0.5 0.7 0.9
P: 20% 20% 20% 20% 20% (Uniform)
AFTER DATA (Posterior):
θ: 0.1 0.3 0.5 0.7 0.9
P: 1% 16% 37% 37% 9% (Peaked at 0.5-0.7)
--------------------------------------------------------------------------
Prior 2 Evolution
BEFORE DATA (Prior):
θ: 1/6 1/2 5/6
P: 33% 33% 33% (Uniform)
AFTER DATA (Posterior):
θ: 1/6 1/2 5/6
P: 6% 62% 32% (Strongly peaked at 0.5)🔙 Back to all notes
𝕏 Let’s Connect
Backlinks:
Machine Learning
