Likelihood Estimation

Here we will be understanding:

1. What Likelihood is?
2. We will apply it to find the best parameters for an already known solution.

Let’s Begin!!

What is Likelihood?

Well there are many good literature around it but still let me make an attempt around it

I assume we are all here aware of Gaussian/Normal distribution. Here is the formula for the same:

Now let’s say we have a Normal Distribution where we have μ = 150cm and σ = 10cm representing the heights of student in a town.

If we need to find the probability of students with height say between 130cm and 150cm. Here is how we can do it in python

from scipy.stats import norm
z_150 = (150 - 150)/10
z_130 = (130 - 150)/10
p = norm.cdf(z_150) - norm.cdf(z_130)
print(round(p * 100,2))

Now interesting thing in Data Science is that what we have is Data.

So, ideally for the above case what we will have is data which is data of heights.

Let’s try to look it from a different point of view.

If we have the data, what would be the the best parameters for our Gaussian Distribution in our case (μ and σ) such that we have the highest likelihood?

As you might have noticed, we have to know the kind of distribution the data follows, in our scenario lets say by using methods like Q-Q plot, Shapiro-Wilks Test we came to know that the height distribution is Normal.

What we do not know is which Normal Curve out of the myriad of possibilities.

Let me rephrase.

If given a pen and paper, we can draw myriad of Normal Distribution curves.

But do we need all these curves?

Not at all!!

We are just concerned with the one we need.

After all I need a shirt which fits me the best right?

In our case the purple curve

How do we get the curve.

Remember each distribution function is defined by a set of parameters. If we figure the parameters, we have our curve. Our only job is to figure out the best parameters μ and σ which will give us the highest values for the distribution function.

Let’s try to put some numbers.

For sometime, lets forget that the μ is 150 and σ is 10.

Below is the data we have for heights column. (For convenience I have taken 4 data points from the whole set)

From Figure-1 we are already familiar with the formula for normal distribution. Now we just need to figure out the μ and σ for this data.

Get Saurav Naik’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Lets start with a random value of μ = 80 and σ = 12

By putting the values into the formula of Normal Distribution function we get the below:

Can you see how small the likelihood values are?

Now one assumption we will make here is that the height data are independent of each other.

And we know that for two independent events P(A ∩ B) = P(A).P(B)

The result you can imagine will be a very very small number. But you can get the idea.

Now don’t you think we could do this very thing with the same data but with different values of μ and σ apart from 80 and 12?

If we had to power to put all such combinations and find out the likelihood for all such cases, the one with the highest likelihood would have been μ = 150 and σ = 10 as I have created the data by sampling the points from the Normal curve with these parameters.

Let’s take a step further and try to prove that μ is actually the mean of the data and σ the standard deviation. For now just think them to be 2 parameters of a function.

All we need to do is find value of μ and σ that yield the maxima (maximum value) of the likelihood function.

We already know that for n data points the likelihood function will look like below:

Now if we come to think about it multiplying all these likelihoods will result in a very very small value. Also multiplication is a more expensive operation than addition.

What we can do is take the log for the hole thing which will make our result look like this:

We will take the first term and continue with the further derivation

Now simplifying it further we will get the below (All I have done is simplifying the above equation using the properties of log):

Now for n such terms we can write the likelihood function as below:

Our objective was to maximize this function.

Let’s first differentiate it w.r.t. μ

making the above equal to 0 we get:

Nice!! can you see that μ is nothing but the mean if we want to maximize the likelihood function

Now let’s differentiate w.r.t. σ

Equating the above to 0 we get:

We see σ is nothing but the standard deviation of the data

Hence, we can conclude that by maximizing the likelihood function we saw that the best parameters for μ and σ are nothing but the mean and standard deviation of the data. That’s true right, we know that from our prior experience with the Normal distribution and now we have proved it as well by using Maximum Likelihood Estimation.