Generative learning

Introduction

So far, we know what linear regression and logistic regression are. Now, in this article, we will take a deep dive into understanding what Generative Learning is?

Index

1. What is Generative Learning?
2. Understanding Bayesian Algorithm As Generative Learning Algorithm?
3. Modelling Input Data Using a Multivariate Gaussian Distribution
4. Modelling Prior Probability Using Bernoulli Distribution
5. Overall Modelling
6. Modelling On Categorical Variables
7. Advantages of Generative Classifiers
8. Advantages of Discriminative Classifiers

1. What is Generative Learning?

Generative learning is a type of machine learning where the model learns how the data is distributed so that it can understand how new data can be generated. Instead of just predicting the probability of an outcome, generative learning tries to model the probability distribution of the data itself.

Example

Let x be an image of animal and y is the label

• Discriminative: model P(y∣x) → Given this input image x, what is the probability of each class y?
• Generative: model P(x∣y) → If the class is y, what kind of input x should we expect?

2. Understanding Bayesian Algorithm As Generative Learning Algorithm?

The Bayesian algorithm can be understood as a generative learning algorithm because it first learns how the input data x is distributed for each class y.

In other words, instead of directly learning P(y∣x), it models P(x∣y), which tells us what the data looks like when it belongs to a particular class. It also learns the prior probability P(y) of each class. Then, using Bayes’ theorem, it combines these probabilities to calculate P(y∣x) and predict the most likely class.

• P(y|x) is Posterior Probability. Probability of the class y given the observed input x.
• P(x|y) is Likelihood. How likely the input x is given a particular class y. This is the part that makes the approach generative because the model learns the distribution of inputs for each class.
• P(y) is Prior Probability: Probability of the class y before seeing the input x.
• P(x) — Evidence / Marginal Probability. Overall probability of observing the input x, considering all possible classes.

3. Modelling Input Data Using a Multivariate Gaussian Distribution

In Bayesian learning, we need to model:

This means: What does the input data x look like when it belongs to class y?

When x contains multiple dimensions, we can model P(x∣y) using a Multivariate Gaussian distribution.

• Multivariate Gaussian Distribution: Probability density function:

• Mean: μ ∈ R^d
• Covariance

4. Modelling Prior Probability Using Bernoulli Distribution

In Bayesian learning, the prior probability is:

It represents the probability of a class before we observe the input x. If the class y has only two possible outcomes, such as:

we can model the class using a Bernoulli distribution.

The Bernoulli probability mass function is:

where:

• θ=P(y=1)
• 1−θ=P(y=0)

5. Overall Modelling

6. Modelling On Categorical Variables

So far, we have considered generative models where the input features x are continuous. For example, we modelled them using a Gaussian distribution.

But what if the input features are discrete or categorical, with no mathematical order?

Let us consider an email classification problem. We want to classify an email as spam or non-spam.

where y=1 represents spam and y=0 represents non-spam.

Representing an email

We can represent an email using a vocabulary of words. Suppose the vocabulary contains d words:

Each word becomes one binary feature:

Therefore,

For example, if the vocabulary is

then an email containing “free offer” can be represented as

This representation is known as multi-hot encoding.

Get Ujjjwalx’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

So each feature is discrete:

The problem with modelling all dependencies

Our goal is to model

If we have d=250000 words, then

Since every feature can be either 0 or 1, there are

possible binary vectors.

If we wanted to model arbitrary dependencies between the words, we would need a probability for every possible binary vector.

Therefore, we would need 2²⁵⁰⁰⁰⁰ probabilities.

Since these probabilities must sum to 1, the number of free parameters would be

This is an astronomically large number. So directly modelling all dependencies between the words is not practical.

The Naive Bayes assumption

We therefore make a strong assumption:

Once we know whether an email is spam or non-spam, we assume that the occurrence of one word does not depend on the occurrence of another word.

Without this assumption, using the chain rule,

The probability of a word can depend on the previous words. With the Naive Bayes assumption, these dependencies are removed:

and

Therefore,

For 50000 features, for example,

This is the key Naive Bayes simplification.

Modelling the Class Variable

The class y is also binary, so we model it using a Bernoulli distribution:

Here, ϕ is the class prior parameter.

Modelling each feature for non-spam emails

For class y=0,

Modelling each feature for spam emails

Thus, the parameters we need to learn are

Learning the Parameters

Now we have defined the model. The next question is: How do we find the values of ϕ, θi,0​, and θi,1​?

We use Maximum Likelihood Estimation (MLE).

Suppose our training dataset is

We want to find the parameters that make the observed training data as likely as possible. Therefore, we define the joint likelihood:

Using the generative decomposition,

Therefore,

Now apply the Naive Bayes assumption:

So,

Using the Bernoulli distributions, this becomes

This is the joint likelihood of the training data.

We then choose the parameters that maximize this likelihood:

In practice, we usually maximize the log-likelihood instead because products of thousands of probabilities can become extremely small:

and find the same optimal parameters by maximizing ℓ.

7. Advantages of Generative Classifiers

• Easy to fit: Generative classifiers are often very easy to fit. For example, a Naive Bayes classifier can be fitted using simple counting and averaging. In contrast, logistic regression requires solving a convex optimization problem.
• Data efficiency: They can often reach stable probability estimates faster than discriminative models when the amount of labelled training data is small.
• Handling missing features: They can easily handle missing features during prediction by simply bypassing the absent features.
• Outlier detection: Generative classifiers can detect outliers because an unusual input has a very low probability under the learned distribution. Thus, the data likelihood P(x) becomes very small.
• Can fit classes separately: Each class distribution can be estimated independently. Therefore, when a new class is added, we can train the model for that new class without retraining the existing class models.

8. Advantages of Discriminative Classifiers

• Better predictive accuracy: Discriminative classifiers are often more accurate than generative classifiers because the conditional distribution p(y∣x) is often much simpler than the joint distribution p(x,y).
• More robust modeling assumption: Some generative classifiers, such as Naive Bayes, make strong independence assumptions that are often not valid. Discriminative classifiers generally make more robust assumptions.
• Can handle feature preprocessing: Discriminative methods allow us to preprocess the input in arbitrary ways. For example, a string of words can be replaced with embedding vectors.