Linear Regression and Logistic Regression in Machine Learning

Understand two important supervised machine learning algorithms, how they work, their mathematical concepts, and the differences between them.

Introduction to Regression in Machine Learning

Machine Learning helps computers learn patterns from data and make predictions without being explicitly programmed for every task.
One important area of Machine Learning is Supervised Learning, where a model learns from labeled data. It uses input features (X) and a target variable (Y) to learn the relationship between them.

Two commonly used supervised learning algorithms are:

• Linear Regression: Used to predict continuous numerical values.
• Logistic Regression: Used to predict categorical outcomes, mainly for classification problems.

For example:

• Predicting house prices based on area, location, and number of rooms is a Linear Regression problem.
• Predicting whether an email is spam or not spam is a Logistic Regression problem.

Although both algorithms have the word “Regression” in their names, they solve different types of problems.
In this article, we will understand how Linear Regression and Logistic Regression work, their mathematical concepts, applications, advantages, disadvantages, and key differences.

What is Linear Regression?

Linear Regression is a supervised machine learning algorithm used to predict continuous numerical values by identifying the relationship between one or more independent variables (input features) and a dependent variable (target).
It works by finding the best-fit straight line through the data points. This line represents the relationship between the input and output variables and helps predict values for new data.

Simple Example

Suppose we want to predict a student’s marks based on the number of hours they study.

As study hours increase, marks also increase. Linear Regression learns this relationship and uses it to predict marks for a student who studies for a particular number of hours.

Important Terms

• Independent Variable (X): The input feature used to make predictions. Example: Study hours.
• Dependent Variable (Y): The output or target we want to predict. Example: Student marks.
• Best-Fit Line: The straight line that represents the relationship between X and Y while minimizing the sum of squared prediction errors.

Applications of Linear Regression

1. Predicting house prices.
2. Estimating salaries based on experience.
3. Predicting sales and revenue.
4. Forecasting product demand.
5. Estimating student marks based on study hours.

How Does Linear Regression Work?

Linear Regression works by identifying the relationship between input features (X) and the target variable (Y). It finds the best-fit line that minimizes the difference between actual values and predicted values.

The algorithm follows these steps:

Step 1: Collect the Data

First, collect a dataset containing input features and the target variable.
Example: Study hours (X) and student marks (Y).

Step 2: Identify the Relationship

The algorithm analyzes how changes in the input variable affect the target variable.

For example, as study hours increase, student marks may also increase.

Step 3: Find the Best-Fit Line

Linear Regression calculates a line that best represents the relationship between the input and output values.

This line is selected by minimizing the squared differences between actual and predicted values.

Step 4: Calculate Predictions

Once the best-fit line is identified, the model uses it to predict the target value for new input data.
For example, if a student studies for 6 hours, the model can estimate their marks based on the learned relationship.

Step 5: Evaluate the Model

Finally, compare predicted values with actual values to understand how well the model performs.

Common evaluation metrics include:

• Mean Absolute Error (MAE)
• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
• R-squared (R²)

Linear Regression Equation

Linear Regression uses a mathematical equation to represent the relationship between independent and dependent variables.
The general equation of Simple Linear Regression is: Y = mX + C

Where:

• Y: Predicted output or dependent variable.
• X: Input feature or independent variable.
• m: Slope or coefficient of the line.
• C: Intercept, the value of Y when X is zero.

Understanding Slope and Intercept

1. Slope (m)

The slope represents how much the predicted output changes when the input variable increases by one unit.

• Positive slope: As X increases, Y increases.
• Negative slope: As X increases, Y decreases.
• Zero slope: The predicted value remains constant.

2. Intercept

The intercept represents the predicted value of Y when X is zero.

Example

Suppose we have the following equation: Y = 10X + 25
If a student studies for 5 hours:
Y = 10(5) + 25
Y = 50 + 25 = 75
Therefore, the predicted marks are 75.

Best-Fit Line

The main objective of Linear Regression is to find the best-fit line that minimizes the difference between actual values and predicted values.
These differences are called residuals or prediction errors.

Types of Linear Regression

Linear Regression is mainly divided into two types:

1. Simple Linear Regression
2. Multiple Linear Regression

1. Simple Linear Regression

Simple Linear Regression uses one independent variable (X) to predict one dependent variable (Y).
It identifies the relationship between a single input feature and the output.
Example: Predicting a student’s marks based on the number of hours they study.

• Independent variable (X): Study hours
• Dependent variable (Y): Student marks

The equation is: Y = mX + C
Here, only one input feature is used for prediction.

2. Multiple Linear Regression

Multiple Linear Regression uses two or more independent variables to predict one dependent variable.
It helps understand how multiple input features influence the target variable.
Example: Predicting house prices based on:

• House area
• Number of bedrooms
• Number of bathrooms
• Location

The general equation is: Y = b0 + b1X1 + b2X2 + ……+ bnXn

Where:

• Y: Predicted output.
• b₀: Intercept.
• b₁, b₂, …, bₙ: Coefficients of the input features.
• X₁, X₂, …, Xₙ: Independent variables.

Difference Between Simple and Multiple Linear Regression

What is Logistic Regression?

Logistic Regression is a supervised machine learning algorithm mainly used for classification problems. It predicts the probability of an input belonging to a particular category or class.
Unlike Linear Regression, which predicts continuous numerical values, Logistic Regression predicts categorical outcomes.
It uses the Sigmoid Function to convert the model’s output into a probability between 0 and 1

Simple Example

Suppose we want to predict whether an email is spam or not spam.
The model analyzes different features of the email, such as:

• Number of suspicious words.
• Number of links.
• Sender information.
• Email content.

Based on these features, Logistic Regression calculates the probability that the email is spam.

For example:

• Probability = 0.85 → Spam
• Probability = 0.20 → Not Spam

Using a classification threshold of 0.5, the model assigns the email to a class.

Types of Problems Solved by Logistic Regression

1. Binary Classification: Predicts between two classes, such as Yes/No or Spam/Not Spam.
2. Multiclass Classification: Predicts one class from more than two categories.
3. Multilabel Classification: Predicts multiple labels for a single input using suitable classification strategies.

Applications of Logistic Regression

1. Email spam detection.
2. Predicting whether a customer will leave a company.
3. Disease prediction (Positive or Negative).
4. Loan approval prediction.
5. Predicting whether a customer will purchase a product.

How Does Logistic Regression Work?

Logistic Regression works by analyzing input features, calculating the probability of an outcome, and assigning the input to a particular class.
It follows these steps:

Step 1: Collect the Data
First, collect a labeled dataset containing input features (X) and a target variable (Y).
Example: A dataset containing email details labeled as Spam or Not Spam.

Step 2: Calculate the Linear Combination
The model combines the input features with their corresponding weights and adds a bias term.
This produces a value that is passed to the Sigmoid Function.

Step 3: Apply the Sigmoid Function
The Sigmoid Function converts the calculated value into a probability between 0 and 1.

For example:

• Probability = 0.85
• Probability = 0.25

These values represent the model’s estimated probability of belonging to the positive class.

Get Gollapalli Sai Teja’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Step 4: Apply the Classification Threshold
The model uses a threshold to convert the predicted probability into a class label.
For a threshold of 0.5:

• Probability ≥ 0.5 → Class 1
• Probability < 0.5 → Class 0

For example:

• 0.85 → Spam
• 0.25 → Not Spam

Step 5: Evaluate the Model
Finally, evaluate the model using classification metrics such as:

• Accuracy
• Precision
• Recall
• F1-Score
• Confusion Matrix

Sigmoid Function in Logistic Regression

The Sigmoid Function is a mathematical function used in Logistic Regression to convert any real-valued input into a probability between 0 and 1.
It produces an S-shaped curve, which is why it is also called the Logistic Function.

Sigmoid Function Formula

The mathematical formula is:

Where:

• σ(z): Output of the Sigmoid Function.
• z: Linear combination of input features and model coefficients.
• e: Euler’s number (approximately 2.718).

The value of z is calculated as:
z=b0+b1X1+b2X2+……+bnXn

How Does the Sigmoid Function Work?

The Sigmoid Function takes the calculated value (z) and converts it into a probability between 0 and 1.

For example:

Important Properties of the Sigmoid Function

1. The output always lies between 0 and 1.
2. It produces an S-shaped curve.
3. When z = 0, the output is 0.5.
4. When z increases, the output approaches 1.
5. When z decreases, the output approaches 0.

Why Do We Use the Sigmoid Function?

Logistic Regression is used for classification, so it needs to estimate the probability of an input belonging to a particular class.
The Sigmoid Function converts the model’s output into a probability that can be used for classification.

For example:

• Probability = 0.90 → Class 1
• Probability = 0.30 → Class 0

Assuming a classification threshold of 0.5

Types of Logistic Regression

Logistic Regression is mainly classified into three types based on the number and nature of output classes.

1. Binary Logistic Regression

Binary Logistic Regression is used when the target variable contains only two possible classes.

Examples:

• Spam or Not Spam
• Yes or No
• Pass or Fail
• Disease Positive or Negative

The model predicts the probability of belonging to one of the two classes using the Sigmoid Function.

2. Multinomial Logistic Regression

Multinomial Logistic Regression is used when the target variable contains three or more unordered categories.

Examples:

• Predicting a student’s preferred subject: Mathematics, Science, or English.
• Predicting a product category: Electronics, Clothing, or Furniture.
• Predicting transport choice: Bus, Train, or Car.

It uses the Softmax Function to calculate probabilities for multiple classes.

3. Ordinal Logistic Regression

Ordinal Logistic Regression is used when the target variable contains three or more categories with a meaningful order.
Examples:

• Customer satisfaction: Low, Medium, High.
• Product ratings: Poor, Average, Good, Excellent.
• Education levels: School, Undergraduate, Postgraduate.

The categories have a natural order, unlike multinomial classification.

Comparison of Logistic Regression Types

Decision Boundary and Classification Threshold

In Logistic Regression, the model predicts the probability of an input belonging to a particular class. To convert this probability into a final class label, we use a Classification Threshold.
A Decision Boundary is the boundary that separates different predicted classes based on the model’s decision rule.

1. Classification Threshold

A classification threshold is a probability value used to decide whether an observation belongs to Class 0 or Class 1.
The default threshold in binary Logistic Regression is usually 0.5.
For example:

• Probability ≥ 0.5 → Class 1
• Probability < 0.5 → Class 0

Consider the following examples:

2. Decision Boundary

A Decision Boundary is a line or surface that separates different classes in the feature space.
For example, suppose we want to classify students as Pass or Fail based on their study hours.

• Students predicted as Pass belong to Class 1.
• Students predicted as Fail belong to Class 0.

The model uses the classification threshold to separate these two classes.

3. Changing the Classification Threshold

The threshold does not always need to be 0.5. It can be adjusted depending on the problem.
For example:
Threshold = 0.5

• Probability ≥ 0.5 → Class 1
• Probability < 0.5 → Class 0

Threshold = 0.7

• Probability ≥ 0.7 → Class 1
• Probability < 0.7 → Class 0

A higher threshold makes the model more conservative about predicting Class 1, while a lower threshold makes it easier to predict Class 1.

Why Is the Threshold Important?

1. It converts probabilities into class predictions.
2. It influences the number of positive and negative predictions.
3. It affects precision and recall.
4. It can be adjusted based on the requirements of the problem.

Linear Regression vs Logistic Regression

Linear Regression and Logistic Regression are both supervised machine learning algorithms. However, they are used for different types of prediction problems.
Linear Regression predicts continuous numerical values, whereas Logistic Regression predicts class probabilities and uses them to perform classification.

Key Differences

Real-World Examples

Linear Regression:
Suppose we want to predict the salary of an employee based on years of experience. The model predicts a numerical value, such as ₹50,000 or ₹75,000.

Logistic Regression:
Suppose we want to predict whether a customer will purchase a product. The model estimates the probability of purchasing and classifies the customer as Yes or No based on a threshold.

When Should We Use Each Algorithm?

Use Linear Regression when:

• The target variable is continuous.
• You want to predict numerical values.
• You want to understand relationships between input features and a numerical output.

Use Logistic Regression when:

• The target variable is categorical
• You want to predict class probabilities
• You want to solve classification problems

13. Advantages and Disadvantages

Both Linear Regression and Logistic Regression are widely used machine learning algorithms. They are simple to understand and useful for solving different prediction problems. However, each algorithm has its own advantages and limitations.

A. Linear Regression

Advantages

1. Easy to Understand: Simple mathematical concept and easy to interpret.
2. Easy to Implement: Can be implemented using Python libraries such as Scikit-learn.
3. Fast Training: Works efficiently on many small and medium-sized datasets.
4. Interpretable Results: Coefficients help understand how input features influence the predicted output.
5. Useful for Numerical Predictions: Suitable for predicting continuous values such as prices, salaries, and sales

Disadvantages

1. Assumes a Linear Relationship: May not perform well when the relationship between input and output is highly non-linear.
2. Sensitive to Outliers: Extreme values can significantly affect the fitted line.
3. Multicollinearity: Highly correlated independent variables can make coefficient estimates unstable.
4. Sensitive to Feature Selection: Irrelevant features can reduce model performance.
5. Underfitting: May fail to capture complex patterns in the data.

B. Logistic Regression

Advantages

1. Easy to Understand: Simple and widely used classification algorithm.
2. Probability Prediction: Estimates the probability of an observation belonging to a class.
3. Fast Training: Efficient for many classification problems.
4. Interpretable: Model coefficients help explain how features influence the predicted outcome.
5. Supports Multiple Classification Types: Can be used for binary, multinomial, and ordinal classification with suitable approaches.

Disadvantages

1. Assumes a Linear Relationship with Log-Odds: May not capture complex non-linear decision patterns without feature transformations.
2. Sensitive to Outliers: Extreme observations can affect model coefficients.
3. Multicollinearity: Highly correlated features can make coefficient estimates unstable.
4. Feature Scaling May Be Helpful: Features with different scales can affect optimization and regularization.
5. Limited for Complex Relationships: May perform poorly when the classes cannot be separated using a suitable linear decision boundary.

14. Conclusion

Linear Regression and Logistic Regression are two important supervised machine learning algorithms used to solve different types of prediction problems.

In this article, we explored how both algorithms work, their mathematical concepts, applications, and differences.

Linear Regression is used to predict continuous numerical values by finding the best-fit line between input features and the target variable.

Logistic Regression is mainly used for classification problems. It uses the Sigmoid Function to calculate probabilities and applies a classification threshold to predict class labels.

Key Takeaways

1. Linear Regression is used for predicting continuous numerical values.
2. Simple Linear Regression uses one independent variable, while Multiple Linear Regression uses two or more independent variables.
3. Mean Squared Error (MSE) measures prediction errors in Linear Regression.
4. Logistic Regression is mainly used for classification problems.
5. Sigmoid Function converts the model’s output into a probability between 0 and 1.
6. Classification Threshold converts predicted probabilities into class labels.
7. Binary, Multinomial, and Ordinal Logistic Regression handle different types of classification problems.
8. Choosing the right algorithm depends on the target variable and the problem requirements.