Linear Regression Algorithm

Introduction

Imagine you are a real estate analyst trying to estimate the price of a house. You notice that larger houses generally cost more than smaller ones. Similarly, a business analyst may observe that sales increase as advertising expenditure rises. These relationships between variables can often be represented using a straight line.

This is where Linear Regression comes into play.

Linear Regression is one of the simplest yet most powerful machine learning algorithms. Despite the rise of complex models such as Random Forests, XGBoost, and Neural Networks, Linear Regression remains a fundamental tool for understanding relationships between variables and making predictions.

In this article, we will explore the concepts behind Linear Regression, understand its mathematical foundation, learn how it works, examine its assumptions, and implement it using Python.

What is Linear Regression?

Linear Regression is a supervised machine learning algorithm used to predict continuous numerical values.

Its primary objective is to find the best-fitting line that describes the relationship between:

• Independent Variable(s) (Features)
• Dependent Variable (Target)

Examples include:

• Predicting house prices
• Estimating sales revenue
• Forecasting stock values
• Predicting employee salaries

The algorithm assumes that a linear relationship exists between input variables and the output variable.

Understanding the Concept

Consider the following data:

Years of ExperienceSalary ($)130,000235,000340,000445,000550,000

As experience increases, salary also increases.

If we plot these points on a graph, they approximately form a straight line.

Linear Regression attempts to find the line that best represents this relationship and uses it to predict future values.

The Equation of Linear Regression

The mathematical equation for Simple Linear Regression is:

[
y = mx + c
]

Where:

• y = Predicted value
• x = Input feature
• m = Slope of the line
• c = Intercept

In machine learning notation:

[
Y = \beta_0 + \beta_1X
]

Where:

• (\beta_0) = Intercept
• (\beta_1) = Coefficient
• (X) = Independent variable
• (Y) = Predicted target value

Understanding Slope and Intercept

Intercept ((\beta_0))

The intercept represents the value of Y when X equals zero.

Example:

If:

[
Salary = 25000 + 5000 \times Experience
]

When Experience = 0:

[
Salary = 25000
]

The intercept is 25,000.

Slope ((\beta_1))

The slope indicates how much the target variable changes when the input variable increases by one unit.

In the previous example:

[
Salary = 25000 + 5000 \times Experience
]

Each additional year of experience increases salary by $5,000.

How Does Linear Regression Work?

The algorithm follows three major steps:

Step 1: Observe Data

Collect historical data containing input and output variables.

Step 2: Fit the Best Line

Determine the line that minimizes prediction errors.

Step 3: Make Predictions

Use the fitted equation to predict unknown values.

What Is the “Best-Fit Line”?

Not every line represents the data equally well.

The best-fit line is the one that minimizes the distance between actual values and predicted values.

These distances are called Residuals.

[
Residual = Actual — Predicted
]

The goal of Linear Regression is to minimize these residuals.

Ordinary Least Squares (OLS)

Linear Regression typically uses the Ordinary Least Squares (OLS) method.

OLS minimizes the sum of squared residuals.

Formula

[
\sum (Actual — Predicted)²
]

Why square the errors?

• Prevents positive and negative errors from canceling out.
• Penalizes larger errors more heavily.
• Simplifies mathematical optimization.

This process determines the optimal slope and intercept.

Types of Linear Regression

1. Simple Linear Regression

Uses one independent variable.

Example

Predicting salary based on years of experience.

[
Y = \beta_0 + \beta_1X
]

2. Multiple Linear Regression

Uses multiple independent variables.

Example

Predicting house price using:

• Area
• Number of Bedrooms
• Location Score
• Age of Property

Equation:

[
Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \dots + \beta_nX_n
]

Multiple Linear Regression is more commonly used in real-world applications.

Assumptions of Linear Regression

For Linear Regression to perform effectively, certain assumptions should be satisfied.

1. Linearity

There should be a linear relationship between features and target.

Good Example

Advertising Spend → Sales

Poor Example

Age → Human Height

Get Sravanichallagondla’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The relationship is not always linear.

2. Independence of Errors

Residuals should not be correlated with one another.

This is particularly important in time-series data.

3. Homoscedasticity

The variance of residuals should remain constant across all predictions.

Desired Pattern

Residuals scattered evenly.

Undesired Pattern

Residual spread increases or decreases systematically.

4. Normal Distribution of Residuals

Residuals should approximately follow a normal distribution.

5. No Multicollinearity

Independent variables should not be highly correlated with each other.

For example:

• Monthly Salary
• Annual Salary

These variables provide similar information.

Evaluating Linear Regression Models

Several evaluation metrics are commonly used.

Mean Absolute Error (MAE)

Measures average absolute prediction error.

[
MAE = \frac{1}{n}\sum |Actual — Predicted|
]

Advantages

• Easy to interpret.
• Less sensitive to outliers.

Mean Squared Error (MSE)

Measures average squared error.

[
MSE = \frac{1}{n}\sum (Actual — Predicted)²
]

Advantages

• Penalizes large errors.

Root Mean Squared Error (RMSE)

Square root of MSE.

[
RMSE = \sqrt{MSE}
]

Provides error in original units.

R-Squared (R²)

Measures how much variation is explained by the model.

[
R² = 1 — \frac{RSS}{TSS}
]

Interpretation

R² ScoreMeaning1.0Perfect Fit0.9Excellent0.5Moderate0.0No Predictive Power

Higher values indicate better explanatory ability.

Implementing Linear Regression in Python

Step 1: Import Libraries

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

Step 2: Load Dataset

data = load_diabetes()

X = data.data
y = data.target

Step 3: Split Data

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

Step 4: Train Model

model = LinearRegression()

model.fit(X_train, y_train)

Step 5: Predict

predictions = model.predict(X_test)

Step 6: Evaluate

mse = mean_squared_error(y_test, predictions)
r2 = r2_score(y_test, predictions)

print("MSE:", mse)
print("R²:", r2)

Output:

MSE: 2900.19
R²: 0.45

Advantages of Linear Regression

1. Easy to Understand

The model is highly interpretable.

2. Fast Training

Requires minimal computational resources.

3. Works Well with Linear Data

Provides excellent results when relationships are linear.

4. Strong Baseline Model

Often used as a benchmark before trying advanced algorithms.

5. Feature Importance Interpretation

Coefficients clearly show feature influence.

Limitations of Linear Regression

1. Assumes Linearity

Cannot effectively model complex nonlinear relationships.

2. Sensitive to Outliers

Extreme values can significantly affect the regression line.

3. Multicollinearity Issues

Highly correlated features reduce reliability.

4. Underfitting Complex Problems

May perform poorly on intricate datasets.

Real-World Applications

Real Estate

Predicting house prices.

Finance

Forecasting stock prices and market trends.

Marketing

Estimating sales based on advertising budgets.

Healthcare

Predicting medical expenses and treatment outcomes.

Human Resources

Estimating salaries based on experience and qualifications.

Linear Regression vs Decision Tree

FeatureLinear RegressionDecision TreeRelationship AssumptionLinearNonlinearInterpretabilityHighHighOutlier SensitivityHighLowComplexitySimpleModerateTraining SpeedFastFastBest Use CaseContinuous Linear DataComplex Decision Rules

Best Practices

• Perform exploratory data analysis before modeling.
• Remove or handle outliers.
• Check for multicollinearity.
• Standardize features when necessary.
• Validate assumptions using residual analysis.
• Use cross-validation for reliable evaluation.

Conclusion

Linear Regression is often the first machine learning algorithm that aspiring data scientists learn, and for good reason. Its simplicity, interpretability, and effectiveness make it one of the most valuable tools in predictive analytics.

By fitting a best-fit line through data points, Linear Regression helps uncover relationships, explain trends, and make informed predictions. While modern machine learning offers more advanced algorithms, Linear Regression remains an essential foundation for understanding statistical modeling and machine learning principles.

Mastering Linear Regression not only improves your ability to build predictive models but also provides the mathematical intuition needed to understand many advanced algorithms used in today’s data-driven world.