Implementing Linear Regression From Scratch using Python

Last Updated : 5 Jun, 2026

Linear Regression is a supervised machine learning algorithm used to predict continuous values by modelling the relationship between input features and output using a best-fit straight line.

  • Predicts continuous outputs such as prices, scores, or sales values.
  • Learns the relationship between independent variables and the dependent variable from training data.
types_of_linear_regression

Types

Linear Regression is mainly of two types:

1. Simple Linear Regression

Simple Linear Regression is a supervised learning algorithm used to predict a continuous target variable using a single input feature. It assumes a linear relationship between the input and output and represents it using a straight line.

Step 1: Import Libraries

Import the required libraries NumPy for numerical operations and Matplotlib for visualization.

R 
import numpy as np
import matplotlib.pyplot as plt

Step 2: Implement Simple Linear Regression Class

Define a SimpleLinearRegression class to model the relationship between a single input feature and a target variable using a linear equation.

  • __init__ method: Initializes slope, intercept, and R² attributes.
  • fit method: Adds bias term, computes slope and intercept using the Normal Equation, and evaluates R² score.
  • predict method: Adds bias to the input X and calculates predicted values using the learned coefficients.
Python 
class SimpleLinearRegression:
    def __init__(self):
        self.coefficient_ = None
        self.intercept_ = None
        self.r2score_ = None
        
    def fit(self, X, y):
        n = len(X)
        X_b = np.c_[np.ones((n,1)), X]  
   
        self.coefficients_ = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
        self.intercept_ = self.coefficients_[0]
        y_pred = X_b.dot(self.coefficients_)
        # R²
        self.r2score_ = 1 - (np.sum((y - y_pred)**2) / np.sum((y - np.mean(y))**2))
        self.y_pred_ = y_pred
        
    def predict(self, X):
        X_b = np.c_[np.ones((len(X),1)), X]
        return X_b.dot(self.coefficients_)

Step 3: Fit the Model and Visualize Results

Fit the model to training data and compute coefficients along with R² score and and plot the data points along with the best-fit regression line.

Python 
X_simple = np.array([1,2,3,4,5,6,7,8,9,10]).reshape(-1,1)
y_simple = np.array([2,4,5,4,5,7,8,9,10,12])

slr = SimpleLinearRegression()
slr.fit(X_simple, y_simple)


print(f"Simple LR Coefficients: {slr.coefficients_}")
print(f"R² Score: {slr.r2score_:.2f}")

plt.scatter(X_simple, y_simple, color='blue', label='Data')
plt.plot(X_simple, slr.y_pred_, color='red', label='Regression Line')
plt.title("Simple Linear Regression")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()
plt.show()

Output:

Screenshot-2026-03-12-171911
Simple Linear Regression

2. Multiple Linear Regression

Multiple Linear Regression is used to predict a continuous target variable based on two or more input features, assuming a linear relationship between the inputs and the output.

Step 1: Import Libraries

Import NumPy for numerical operations, Matplotlib for plotting and mpl_toolkits.mplot3d to create 3D visualizations.

Python 
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

Step 2: Implement Multiple Linear Regression Class

Here we implement a Multiple Linear Regression class to model the relationship between multiple input features and a continuous target variable using a linear equation.

  • __init__ method: Initializes coefficients (slopes), intercept (bias), and R² score for model evaluation.
  • fit method: Adds bias term, computes coefficients using the Normal Equation, predicts outputs, and calculates R² score.
  • predict method: Adds bias to input data and generates predictions using learned coefficients.
Python 
class MultipleLinearRegression:
    def __init__(self):
        self.coefficients_ = None 
        self.intercept_ = None     
        self.r2score_ = None    
        
    def fit(self, X, y):
        n = X.shape[0]
        X_b = np.c_[np.ones((n,1)), X]  
        self.coefficients_ = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
        self.intercept_ = self.coefficients_[0]
        y_pred = X_b.dot(self.coefficients_)
        self.r2score_ = 1 - (np.sum((y - y_pred)**2)/np.sum((y - np.mean(y))**2))
        self.y_pred_ = y_pred
        
    def predict(self, X):
        X_b = np.c_[np.ones((X.shape[0],1)), X]  
        return X_b.dot(self.coefficients_)

Step 3: Generate Sample Dataset

We create a small dataset with two input features and a target variable, adding noise to simulate real-world data.

Python 
np.random.seed(0)
X1 = np.random.randint(1, 11, 15)
X2 = np.random.randint(1, 11, 15)
X_multi = np.column_stack((X1, X2))
y_multi = 1 + 2*X1 + 3*X2 + np.random.randn(15)*2

Step 4: Fit the Model and Visualize

We train the Multiple Linear Regression model on the dataset, print coefficients and R² score, and visualize the regression plane in 3D.

Python 
mlr = MultipleLinearRegression()
mlr.fit(X_multi, y_multi)
print(f"Multiple LR Coefficients: {mlr.coefficients_}")
print(f"R² Score: {mlr.r2score_:.2f}")

fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(111, projection='3d')

ax.scatter(X_multi[:,0], X_multi[:,1], y_multi, color='blue', label='Data')

x1_surf, x2_surf = np.meshgrid(
    np.linspace(X_multi[:,0].min(), X_multi[:,0].max(), 10),
    np.linspace(X_multi[:,1].min(), X_multi[:,1].max(), 10)
)

pred_surf = mlr.predict(np.c_[x1_surf.ravel(), x2_surf.ravel()]).reshape(x1_surf.shape)

ax.plot_surface(x1_surf, x2_surf, pred_surf, color='red', alpha=0.5, rstride=1, cstride=1)

ax.set_xlabel('X1')
ax.set_ylabel('X2')
ax.set_zlabel('y')
ax.set_title("Multiple Linear Regression with Regression Plane")
ax.legend()
plt.show()

Output:

Screenshot-2026-03-12-173056
Multiple linear regression

Download code from here.

Comment

Explore