Bayesian Neural Networks : An Introduction

What is Bayesian Inference? Understanding Prior, Likelihood, and Posterior

Bayesian inference is a statistical method that involves updating the probability of a hypothesis as more evidence or information becomes available. This process uses Bayes’ theorem, a fundamental principle in probability theory, which relates the conditional and marginal probabilities of random events.

Bayesian inference allows us to incorporate prior beliefs, update these beliefs as new data is observed, and arrive at a posterior belief that reflects both the prior and the new data. Let’s break down the key components of Bayesian inference: prior, likelihood, and posterior.

Prior, Likelihood, and Posterior: The Building Blocks of Bayesian Inference

1. Prior p(theta)
The prior represents our initial belief about the parameter (theta) before seeing any data. It is a probability distribution that encapsulates our existing knowledge or assumptions. For example, if we are estimating the probability of a coin landing heads, we might assume a uniform prior ( p(theta) = 0.5 ), representing a fair coin.

2. Likelihood p(D \theta)
The likelihood quantifies the probability of observing the data ( D ) given a specific parameter value (theta). It measures how well a particular parameter explains the observed data. In our coin example, if we observe 7 heads in 10 flips, the likelihood of the data given that the coin is fair ( theta = 0.5 ) can be…