Applied Machine Learning — Part 20 Introduction to Mean Field Inference

mean field inference

Mean Field Inference (MFI) is a powerful technique used in statistical physics and machine learning to approximate complex probability distributions. It simplifies the problem of inference in probabilistic models by assuming that the dependencies between variables can be decoupled. This method is particularly useful when dealing with large datasets or models where exact inference is computationally prohibitive.

In this blog post, we’ll delve into the basics of Mean Field Inference, walk through a Python implementation, and create some fancy visualizations to illustrate the concept.

Understanding Mean Field Inference

Mean Field Inference works by approximating a complex joint distribution with a product of simpler distributions. The key idea is to replace the joint distribution P(X) with a factorized distribution Q(X)=∏i​Qi​(Xi​), where Qi​(Xi​) are simpler, individual distributions over each variable Xi​.

The goal is to find the Qi​ that best approximates the true distribution by minimizing the Kullback-Leibler (KL) divergence between P and Q:

KL(Q∣∣P)=∑X​Q(X)logP(X)Q(X)​

Get Connie Zhou’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

By minimizing this divergence, we can make the factorized distribution Q as close as possible to the true distribution P.

Python Implementation of Mean Field Inference

Let’s implement Mean Field Inference in Python using a simple example. We’ll consider a binary Markov Random Field (MRF) where each variable can take on the values 0 or 1.

Step 1: Define the MRF

We’ll start by defining the energy function of our MRF. For simplicity, we’ll use a pairwise MRF with binary variables.

import numpy as np

# Number of variables
n_vars = 5

# Randomly initialize pairwise potentials
np.random.seed(42)
pairwise_potentials = np.random.randn(n_vars, n_vars)

# Ensure symmetry
pairwise_potentials = (pairwise_potentials + pairwise_potentials.T) / 2

# Define unary potentials
unary_potentials = np.random.randn(n_vars)

Step 2: Initialize the Factorized Distribution

We’ll initialize the factorized distribution Qi​(Xi​) as independent Bernoulli distributions.

# Initialize mean field factors (probabilities of each variable being 1)
q = np.random.rand(n_vars)

# Ensure probabilities are between 0 and 1
q = np.clip(q, 0.1, 0.9)

Step 3: Mean Field Update Equations

We’ll iteratively update the factors using the mean field update equations until convergence.

def mean_field_update(q, unary_potentials, pairwise_potentials, n_iters=100, tol=1e-3):
    for _ in range(n_iters):
        q_new = np.copy(q)
        for i in range(n_vars):
            q_i = unary_potentials[i]
            for j in range(n_vars):
                if i != j:
                    q_i += pairwise_potentials[i, j] * q[j]
            q_new[i] = 1 / (1 + np.exp(-q_i))
        
        # Check for convergence
        if np.max(np.abs(q_new - q)) < tol:
            break
        q = q_new
    return q

# Perform mean field inference
q = mean_field_update(q, unary_potentials, pairwise_potentials)
print("Final probabilities:", q)

Visualizing the Results

To better understand the results, we’ll create visualizations of the factorized distribution and the inferred probabilities.

Visualization with Matplotlib

We’ll use Matplotlib to create a heatmap of the pairwise potentials and a bar chart of the inferred probabilities.

import matplotlib.pyplot as plt
import seaborn as sns

# Heatmap of pairwise potentials
plt.figure(figsize=(8, 6))
sns.heatmap(pairwise_potentials, annot=True, cmap='coolwarm')
plt.title('Pairwise Potentials')
plt.show()

# Bar chart of inferred probabilities
plt.figure(figsize=(8, 6))
plt.bar(range(n_vars), q, color='skyblue')
plt.xlabel('Variable')
plt.ylabel('Inferred Probability')
plt.title('Inferred Probabilities for Each Variable')
plt.show()

Conclusion

In this blog post, we’ve explored the basics of Mean Field Inference, implemented it in Python, and visualized the results. Mean Field Inference is a versatile tool that can be applied to various probabilistic models, making it a valuable technique in both machine learning and statistical physics. With the provided code, you can experiment with different MRF structures and potentials to see how Mean Field Inference performs in different scenarios.

Feel free to extend this example to more complex models and share your findings. Happy coding!