Probabilistic graphical models or PGM are frameworks used to create probabilistic models of complex real world scenarios and represent them in compact graphical representation. This definition in itself is very abstract and involves many terms that needs it’s own space, so lets take these terms one by one.

Model
A Model is a declarative representation of a real world scenario or a problem that we want to analysis. When I say declarative it means it’s not derived but declared or defined either by a domain expert using its domain knowledge of the problem or by using statical learning algorithms from historic data set, and then represented using any mathematical tools like graph or even simply by an equation.
For ex in Liner Regression

Why we need to model the problem? Because of two main reasons
1. It allows us to translate an unstructured real world problem in to a structured mathematical representation
2. It allows us to isolate the problem (model representation) with it’s solution (algorithm), meaning once we have a mathematical model for our problem we can apply any algorithm to solve it, e.g. for the above model we can either solve it using gradient decent or any other method, model doesn’t change.

Probabilistic
The nature of the problem that we are generally interested to solve or the type of queries we want to make are all probabilistic because of uncertainty. There are many reasons that contributes to it. For example, incomplete knowledge of the problem, noisy observations or some attributes that contributes to the problem but can’t/hasn’t included in the model.

Graphical
In the example above we used a mathematical representation of the model but in real world scenarios are complex and often involves large no of variables. So many times a graphical representation helps us to visualise better and then we use Graph Theory to reduce the no of relevant combinations of all the participating variables to represent the high dimensional probability distribution model more compactly.

There are some basic terms & concepts related to probability and statistics I will be using to discuss PDMs. Those are covered in a separate post Basic Probability Theory and Statistics.

A model is generally useful if it helps us to understand the real world phenomena allows us to make useful predictions about how the world will behave under certain set of conditions. The kind of quires we often want to do on our models are of nature is to predict what will likely to happen when something else is already happened/observed. To answer such queries we need to consider all components (random variables) of that state of the world and represent them some how. We already learned “joint distribution”, we represents all possible outcomes of an event. So it means we got the tool to represent our model which can cover all possible scenarios and that is ‘Joint Probability Distribution Table’ (For time being restrict ourself to discrete random variables only). Then why we are discussing PGMs?
The problem is even for a simple model the number of variables n can be in the range of 100s and if each can obtain on an average d values, the size of the joint distribution table can be very-2 large. So tables may not be a feasible way to represent complex models though the conceptually they are sufficient, computational limitation don’t allow them. We need some other representation, and it seems Graphs because of it’s sparse nature can be a good fit. There are number of advantages of using graphs to represent such probabilistic models.

1. Even though there are large number of random variables involved not necessarily they all depends on each other. And if the two random variables are independents we don’t need to consider their all combinations in join distribution table, which means ideally there will be lesser rows in the table. Graph Theory has inherent capability to represent such dependence and independences.
2. Graphs representation is flexible in a sense that one doesn’t require to obtain all the knowledge about world to build a model. One can start with his understanding of the world and build a model and it’ll behave based on the knowledge it has. When one acquire more knowledge it can apply the same incrementally to the model (Add/Update nodes, edges) and model will produce improved results based on new information.
3. As graph are standard mathematical structure it not only allows to encode probability distribution but also provide a very clear interface to query model for prediction queries.

As PGMs are based on Graph Theory they can be categories in to two categories as Directed Graphs or Undirected Graphs. There is one family of PGM uses Directed Graphs is Bayesian Networks and other that uses undirected graphs is Markov Networks. Even though they both derived from Graph Theory there are so many differences in both of their intuitions and each require it’s own space ‘ll discuss them in separate posts.

Probabilistic Graphical Models — Introduction was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Generative Learning Algorithms:
In Linear Regression and Logistic Regression both we modelled conditional distribution of y given x, as follow.

Algorithms that model p(y|x) directly from the training set are called discriminative algorithms.
There can be a different approach to the same problem, consider the same binary classification problem where we want learn to distinguish between two classes, class A (y=1) and class B (y=0) based on some features. Now we take all the examples of label A and try to learn the features and build a model for class A. Then we take all the examples labeled B and try to learn it’s features and build a separate model for class B. Finally to classify a new element, we match it against each model and see which one fits better (generate high value for probability). In this approach we try to model p(x|y) and p(y) as oppose to p(y|x) we did earlier, it’s called Generative Learning Algorithms.
Once we learn the model p(y) and p(x|y) using training set, we use Bayes Rule to derive the p(y|x) as

Gaussian discriminant analysis model
When we have a classification problem in which the input features are continuous random variable, we can use GDA, it’s a generative learning algorithm in which we assume p(x|y) is distributed according to a multivariate normal distribution and p(y) is distributed according to Bernoulli. So the model is

Now as we did in Linear Regression and Logistic Regression, we need to define the log likelihood function L and then by maximising L with respect to model parameters, find the maximum likelihood parameters.

So Eq(2) , Eq(4) and Eq(5) defines all the maximum likelihood parameters of GDA as below

Example: I used a custom iris dataset (reduced features to fit it in to two dimensions and removed the 3rd class from the data set to make it for binary classification)for this example.The data plot looks as

For the dataset computed model parameters are as below

Based on the algorithm the probability density plots for each models look as

And from their contour plots you can see the models explains the test data. The surface of the models are not smooth (As the assumption p(x|y) is guassian is not perfectly true for this particular test data. And will discuss how it affects the accuracy of the model) and so the contours will also not be smooth but noisy.

Lets take two test points to test our algorithm as
x0 = [4, 4]
x1 = [6.5, 2.25]

You can see the algorithm favours the class 0 for x0 and class 1 for x1 as expected. Both Logistic Regression and Gaussian Discriminant Analysis used for classification and both will give a slight different Decision Boundaries so which one to use and when.
GDA makes an assumption about the probability distribution of the p(x|y=k) where k is one of the classes. And it can be easily proved that for GDA, if the initial assumptions about distribution of p(x|y=k) (Gaussian)and p(y)(Bernoulli) are true, then p(y|x) can be expressed as Sigmoid. But vice-versa is not true. Which means GDA makes more specific assumptions about the data set then Logistic Regression and if those assumptions are true then it works better then LR. This fact can be useful when you know better about the nature of the dataset you’re dealing with and which is Gaussian, and you don’t have a large training set, in situations like this GDA will perform better then LR. But on the other hand LR makes more generic assumptions and can be more useful in lot of other places where the probability distribution of the feature set is not Gaussian.

Gaussian Discriminant Analysis an example of Generative Learning Algorithms was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is also know as ‘Classification’. This is used in many scenarios where we want to categories input in predefined classes. For ex. tag email as spam/non spam, predict the age group of a customer from the data of commerce portal etc.
In linear regression the output domain is a continues range, i.e. it’s a infinite set, while in logistic regression the output y we want to predict takes only a small no of discrete values. i.e. it’s Finite Set. For simplicity lets consider a binary classification where y can take only two values, 1 (positive) and 0 (negative).
Just like linear regression we need to start with a hypothesis. As the output domain is bounded (0,1) it doesn’t make sense to have a hypothesis which produces value beyond this range.

Given the above set of logistic regression models (why set? because theta is variable) we need to find the co-efficient theta for the best fit model which best explains the training set. For that we need to start with a set of probabilistic assumptions parameterised by theta and then find the theta via Maximum Likelihood.
Lets start with Bernoulli distribution , the probability distribution of a random variable which takes the value of 1 with probability p and value 0 with probability q= 1-p.

In linear regression we find the coefficients by equating the derivative of log likelihood to zero. We evaluated the derivative of likelihood just like we did but the resultant Ex(3) is not a mathematically closed equation that we can solve. (Remember x and theta both are vectors in the eq and h is a non linear function)
We can still find the coefficient by using a brute force algorithm called Gradient Ascent. where we start with some coefficient and then keep updating theta iteratively until the likelihood function converges.

Example
Let’s take the wikipedia example
Suppose we wish to answer the following question:

A group of 20 students spend between 0 and 6 hours studying for an exam. How does the number of hours spent studying affect the probability that the student will pass the exam?

Logistic Regression was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

linear regression is a linear approach for modeling the relationship between a scalar dependent variable y and one or more or independent variables denoted X. Unlike Logistic Regression the dependent variable here can take a continues range of values. The relationships are modeled using linear predictor functions called Linear Hypothesis, whose unknown model parameters are estimated from the data.
Lets start with a linear hypothesis.

Where

We can write the same in concise notation as below.

The above equation signifies that given a new input feature vector x the predicted value of the output will be given by the hypothesis h. So the accuracy of the predictions depends on the weights of the input features.

If we denote actual output values for the m training examples by a vector y of size m. One reasonable way is to use training set to make value of h as close to y. We define the actual difference as cost function J given by.

So now the equation becomes find weights for which cost function J is minimum.

Let

Where
Y : is output vector for m training examples.
X : is matrix of size m*n where each ith row belongs to ith training set.
Q : is weight vector of size n for n training features.
So

And

Which results In

Example:
Given Training set here
X = number of claims
Y = total payment for all the claims in thousands of Swedish Kronor

The above will Resolve to

And Finally the model will be

Linear Regression was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Definition:
matrix calculus is a specialised notation for doing multivariable calculus. It collects the various partial derivatives of a single function with respect to many variables, and/or of a multivariate function with respect to a single variable, into vectors and matrices that can be treated as single entities.
The six kinds of derivatives that we are going to discuss can be organised in a matrix as below.

Scalar By Scalar:
If y is defined as continuous and differentiable function of x for the range (x1, x2) such that.

And it’s equal to the slope of function y at point x=t. It can also be understood as rate of change y w.r.t to infinitely small change in x.
Ex:

Vector by Scalar:

Ex:

Matrix by Scalar:

Ex:

Scalar By Vector:
Let y is a scalar function of n independent variables such that

Then the derivative of y w.r.t to vector x is given by the vector equation

Ex:

Vector By Vector:

Ex:

Scalar By Matrix

Ex:

Matrix
The ordered rectangular array

Is called a Matrix of order m*n
where i=1,2,3… m and j=1,2,3… n.

Vector
A vector is a matrix with only one column.

Note:
Multicolumn matrices are denoted by bold uppercase alphabets like A, B … Z
Vectors are denoted by bold small case letters like alphabets like a, b… z

Null Matrix
Null matrix is a matrix all elements 0s. It’s denoted by Z and defined as

Equal Matrix
Let

And

Then A=B if and only if they are of same order and all corresponding elements are also equal i.e. n=q and m=p and

Identity Matrix
Identity matrix is a square matrix with 1s on all the main diagonal positions and 0s every where else. It’s denoted by I and defined as

Ex:

Matrix Addition
The addition of two matrix is only defined if they are of same order.
Let
A(m * n) & B(m * n)
Then it’s summation C = A + B is defined as

For all i = 1, 2, 3 … m j = 1, 2, 3 … n
C’ll be m * n matrix

Matrix Scalar Multiplication
Let

Then their product is

is a matrix B of order m*n whose elements are defined as

Matrix Multiplication
Two matrices are multiplicable if and only if they are of orders A(m*n) and B(n*p).

And it’s product C = AB is defined as

For all i = 1, 2, 3 … m j = 1, 2, 3 … p.
C’ll be m * p matrix

Matrix Transpose
Transpose of a matrix A is denoted as

and is defined as a matrix whose elements are given by eq

Matrix Trace
Trace of a matrix is defined only for square matrix. Let A is (n*n). Then trace of the matrix A, tr(A) is defined as sum of all the elements on it’s main diagonal.

SubMatrix
Let A is m*n matrix. Then the sub matrix is (m-1)*(n-1) matrix obtained from A by removing row i and column j.
Ex

Then sub matrix

Matrix Determinant
Determinant of a matrix is only defined for square matrix.
Let A is square matrix
Then determinant of A is denoted as det(A) or |A| and is defined as

Cofactor Matrix
Cofactor matrix is defined for only square matrix.
Let A is n*n matrix. Then cofactor matrix C of A is defined as

Adjugate of a Matrix
Adjugate of a matrix is defined only for square matrix.
Let A is n*n matrix. Then Adjugate of A is denoted by adj(A) and is defined as.

Inverse of Matrix
Inverse of a matrix is defined for only for square matrix.
Let A is (n*n) matrix
Inverse of a matrix is denoted as

And is defined as

Where I is an identity matrix of order n. A is invertible if and only if

And is evaluated by the equation