Bayesian inference aims to compute the posterior distribution over the model parameters w given the data D:
Press enter or click to view image in full size
Why is this Integral Intractable?
For deep neural networks, this integral is computationally intractable due to:
1️. High-Dimensional Parameter Space
Deep networks have millions of parameters w.
Computing the posterior requires integrating over all possible weight values.
This is exponentially complex and cannot be done analytically.
2️. No Closed-Form Solution
Bayesian inference in simple cases (like Gaussian conjugate priors) has closed-form solutions.
However, deep networks have nonlinear activations (ReLU, sigmoid, etc.), making the integral non-analytical.
3️. Dependence Between Parameters
In deep networks, weight updates are highly correlated across layers.
This correlation prevents factorizing the posterior into independent components, making exact inference harder.
4️. Expensive Marginalization Over All Parameters
Exact Bayesian inference requires evaluating every possible configuration of weights.
For a deep network, this means solving an integral over a high-dimensional space — which is practically impossible.
🚀 How Do We Approximate Bayesian Inference?
Since exact inference is intractable, we approximate it using:
Get Anuj shah (Exploring Neurons)’s stories in your inbox
Join Medium for free to get updates from this writer.
✅ Variational Inference (VI): Approximate P(w∣D) with a simpler distribution q(w∣θ) and minimize KL divergence. ✅ Monte Carlo Sampling (MCMC, SGLD): Estimate posterior using stochastic sampling methods. ✅ Laplace Approximation: Approximate posterior with a Gaussian around the MAP estimate. ✅ Bayesian Neural Networks (BNNs): Use reparameterization and VI to learn an approximate posterior.
