Multiple Linear Regression

Linear Regression

Linear regression is a statistical technique used to find the relationship between variables. In ML context, linear regression finds the relationship between independent variables (features) and a dependent variable (label).

Multiple Linear regression is a kind of linear regression in which there are multiple features and a single label.

Potatoes yield Analogy :

To predict the yield on the basis of fertilisers, amount of rain and sunlight. These factor affect the potato yield therefore the equation will contain all these factors.

In multiple linear regression the scatter plot of dataset is at least in 3 Dimensional space and can go up to n dimension depending upon the number of features.

Understanding with 3D MLR :

In linear regression we were trying to find a relation between x and y coordinates, in 3D MLR we have two features (say x and y) and a label (z) :

· A scatter plot between three column will be a graph with points distributed in 3D space

· To find a relationship between three columns we have to find a plane in 3D space.

In higher dimension this plane is known as a hyperplane.

Equation of Hyperplane:

· Beta is the offset

· If B1>B2 This implies prediction depends more on X1 than X2

· Lower the values of coefficients lower will be the dependency on feature/independent variable.

Get Shahbaz Ali’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To practise we can generate our own regression data set using make_regression

Mathematical formulation of Multiple LR equation :

Let’s consider a 4D dataset (of 100 examples) which include Cgpa, IQ, gender and package as variables.

Each of these example will have a predicted values according to our model/equation, this equation gives the predicted value by using the particular dependent variables. Therefore each of these predicted values can be represented along their corresponding feature values as follows :

Let’s assume instead of 100 examples we have n examples with m features :

Rewriting the matrix on RHS as a product of two matrix i.e. X(with extra column of 1’s) and Beta :

Therefore we write matrix as the product of two :

To prove the matrix as same we find the resultant shape of matrix and it comes out to be a matrix.

With this we calculated matrix of coefficient such that the plane formed gives minimum error but there is one problem in using OLS method, as the calculated inverse involves cubic complexity. Therefore the computation increases cubically with increase in the sample.

For this reason we don’t use this techniques and instead go for approximation using Gradient Descent

Implementation of OLS method with above formula:

class MeraLR:
def __init__(self):
        self.coef_ = None
        self.intercept_ = None

    def fit(self, X_train, y_train):
        X_train = np.insert(X_train, 0, 1, axis=1)

        # calcuate the coeffs
        betas = np.linalg.inv(np.dot(X_train.T, X_train)).dot(X_train.T)
                                                                .dot(y_train)
        self.intercept_ = betas[0]
        self.coef_ = betas[1:]

    def predict(self, X_test):
        y_pred = np.dot(X_test, self.coef_) + self.intercept_
        return y_pred

In the next post lets explore more about Gradient Descent.