Logistic Regression: A Comprehensive Guide to Understanding the Machine Learning Algorithm

1. Introduction

Logistic Regression is one of the most widely used supervised machine learning algorithms for solving classification problems. Despite having the word regression in its name, it is primarily used to predict categorical outcomes rather than continuous numerical values.

In machine learning, classification means assigning data points to predefined categories. For example:

• Predicting whether an email is spam or not spam.
• Predicting whether a customer will purchase a product.
• Predicting whether a student will pass or fail an examination.
• Predicting whether a patient has a particular disease based on medical test results.

Logistic Regression helps us estimate the probability that an observation belongs to a particular class.

2. What Is Logistic Regression?

Logistic Regression is a supervised learning algorithm used to predict the probability of a categorical dependent variable based on one or more independent variables.

It works by applying the logistic (sigmoid) function to a linear combination of input features. This converts the output into a probability between 0 and 1.

For example, suppose a bank wants to predict whether a customer will repay a loan.

The input features might include:

• Customer income
• Age
• Credit score
• Employment status
• Existing debt

The model analyzes these features and predicts the probability that the customer will repay the loan.

If the predicted probability is greater than or equal to 0.5, the model may classify the customer as likely to repay. Otherwise, it may classify the customer as unlikely to repay.

3. How Does Logistic Regression Work?

Logistic Regression works in two major steps.

Step 1: Calculate a linear combination

First, the algorithm calculates a weighted sum of the input features.

z = b0 + b1x1 + b2x2 + . . . .+ bnxn

Where:

• Z = Linear combination of inputs
• b0 = Intercept or bias
• b1 , b2 , . . . .bn = Model coefficients
• x1, x2 , . . . ,xn = Input features

Step 2: Apply the sigmoid function

The calculated value is passed through the sigmoid function.

The sigmoid function converts any real-valued input into a probability between 0 and 1.

The probability can then be converted into a class prediction using a threshold.

For example:

• Probability ≥ 0.5: Class 1
• Probability < 0.5: Class 0

The threshold can be adjusted depending on the requirements of the problem.

4. Types of Logistic Regression

There are three major types of Logistic Regression.

A. Binary Logistic Regression

Binary Logistic Regression is used when the dependent variable has only two possible outcomes.

Examples:

• Yes or No
• Pass or Fail
• True or False
• Spam or Not Spam

B. Multinomial Logistic Regression

Multinomial Logistic Regression is used when the dependent variable has more than two categories, and those categories do not have a natural order.

Examples:

• Predicting whether a customer prefers a car, bike, or bus.
• Classifying news articles as sports, politics, or entertainment.

C. Ordinal Logistic Regression

Ordinal Logistic Regression is used when the dependent variable has more than two categories with a meaningful order.

Get T.kiran’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Examples:

• Customer satisfaction: Low, Medium, High.
• Education level: School, Undergraduate, Postgraduate.

5. Example of Logistic Regression

Let’s consider a simple example of predicting whether a student will pass or fail based on the number of hours spent studying.

Suppose a student studies for 5 hours, and the model predicts a passing probability of 0.85.

Since 0.85 is greater than 0.5, the model predicts:

Result: Pass (Class 1).

This demonstrates how Logistic Regression uses input data to estimate the probability of a categorical outcome.

6. Cost Function in Logistic Regression

Logistic Regression uses a loss function called Log Loss or Binary Cross-Entropy Loss.

Unlike Linear Regression, which commonly uses Mean Squared Error, Logistic Regression uses Log Loss to measure how far the predicted probabilities are from the actual class labels.

The formula is:

Where:

• m = Number of training examples
• yi= Actual class label
• ^ yi = Predicted probability

The objective of the algorithm is to minimize the loss function so that its predictions become more accurate.

The model commonly uses an optimization technique called Gradient Descent to update its coefficients and minimize the loss.

7. Evaluation Metrics for Logistic Regression

After training a Logistic Regression model, we need to evaluate its performance.

Some commonly used evaluation metrics are:

1. Accuracy

Accuracy measures the percentage of correctly predicted observations.

2. Precision

Precision measures how many predicted positive cases are actually positive.

3. Recall

Recall measures how many actual positive cases are correctly identified.

4. F1-Score

F1-score is the harmonic mean of precision and recall.

5. Confusion Matrix

A confusion matrix shows the number of correct and incorrect predictions made by the classification model.

8. Advantages of Logistic Regression

1. Simple to understand: It is easy to implement and interpret.
2. Computationally efficient: It generally requires less training time than complex machine learning algorithms.
3. Probability prediction: It provides probabilities rather than only class labels.
4. Works well for binary classification: It is particularly useful for problems involving two categories.
5. Interpretable coefficients: Model coefficients help explain how input variables influence the predicted odds.

9. Disadvantages of Logistic Regression

1. Linear decision boundary: Standard Logistic Regression assumes a linear relationship between input features and the log-odds of the outcome.
2. Sensitive to multicollinearity: Highly correlated input variables can make coefficient estimates unstable.
3. Feature engineering: Complex relationships may require additional feature transformations.
4. Outlier sensitivity: Extreme observations can influence the estimated coefficients.
5. Limited for complex patterns: It may not perform well when the relationship between variables is highly nonlinear unless suitable transformations are introduced.

10. Conclusion

Logistic Regression is an important and fundamental algorithm in machine learning. Although its name contains the word regression, it is mainly used for classification tasks.

It works by combining input features using a linear equation and transforming the result into a probability through the sigmoid function. Based on a selected threshold, the model assigns observations to different classes.

Because of its simplicity, efficiency, interpretability, and ability to estimate probabilities, Logistic Regression is widely used in healthcare, banking, marketing, education, and many other fields.

Understanding Logistic Regression provides a strong foundation for learning more advanced classification algorithms such as Decision Trees, Random Forests, Support Vector Machines, and Neural Networks.