Logistic Regression

Logistic Regression is a supervised machine-learning algorithm used for binary classification problems. Unlike linear regression, which predicts continuous values, it predicts the probability that an input belongs to a particular class such as 0/1, Yes/No, or True/False.

It uses the sigmoid (logistic) function to convert linear input values into a probability between 0 and 1, producing an S-shaped curve. A threshold value (commonly 0.5) is then applied to classify the input into one of the two classes.

Types of Logistic Regression

Logistic regression is categorized based on the type of output variable it is used to predict. There are three main types:

1. Binomial Logistic Regression

Binomial logistic regression is used when the output variable has only two possible outcomes. These outcomes are usually represented as 0 and 1, such as Yes/No, Pass/Fail, or True/False. It is the most commonly used type and is mainly applied in binary classification problems.

2. Multinomial Logistic Regression

Multinomial logistic regression is applied when the output variable has more than two categories and no natural order among them. Each category is treated independently. For example, predicting an animal type such as cat, dog, or sheep.

3. Ordinal Logistic Regression

Ordinal logistic regression is used when the output variable has multiple categories that follow a specific order or ranking. The model considers this order while making predictions. Examples include categories like low, medium, and high or rating scales.

How does Logistic Regression work?

Logistic regression model transforms the linear regression function continuous value output into categorical value output using a sigmoid function which maps any real-valued set of independent variables input into a value between 0 and 1. This function is known as the logistic function.

Suppose we have input features represented as a matrix:

At this stage, z is a continuous value from the linear regression. Logistic regression then applies the sigmoid function to z to convert it into a probability between 0 and 1 which can be used to predict the class.

Now we use the sigmoid function where the input will be z and we find the probability between 0 and 1. i.e. predicted y.

Logistic Regression Equation and Odds

Logistic regression predicts the probability that a given input belongs to a particular class (usually Class 1). Instead of modeling the probability directly, it first models the odds of the event occurring.

1. Odds in Logistic Regression

The odds represent how likely an event is to occur compared to it not occurring. They are defined as:

where:

• p(x) is the probability that the output belongs to Class 1
• 1−p(x) is the probability that it belongs to Class 0

2. Log-Odds (Logit)

To make the model linear, we take the natural logarithm of the odds. This is called the log-odds or logit:

The value zzz is expressed as a linear combination of input features:

z=w⋅x+b

where:

• w represents the weights
• x represents the input features
• b is the bias term

3. Deriving the Logistic Regression Equation

By exponentiating both sides of the log-odds equation and rearranging terms, we obtain the probability expression:

This can also be written as:

4. Final Interpretation

This final equation is the logistic regression formula. It converts the linear score w⋅X+bw \cdot X + bw⋅X+b into a value between 0 and 1, representing the probability that the input belongs to Class 1.

Get Shakeraafreen’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If this probability is greater than or equal to a chosen threshold (usually 0.5), the input is classified as Class 1; otherwise, it is classified as Class 0

Terminologies Involved in Logistic Regression

• Independent Variables (Features):
  These are the input values used by the model to make predictions, such as age, size, or test measurements.
• Dependent Variable (Target):
  This is the output variable that the model predicts. In logistic regression, it is categorical, usually binary (0 or 1).
• Logistic (Sigmoid) Function:
  This function converts the linear combination of inputs into a value between 0 and 1, which represents probability.
• Odds:
  Odds express how likely an event is to happen compared to it not happening. It is calculated as

• Log-Odds (Logit):
  This is the natural logarithm of odds. Logistic regression models this log-odds as a linear equation of the input features.
• Coefficients (Weights):
  These values show how much each independent variable influences the final prediction.
• Intercept (Bias):
  This is the constant term in the model and represents the log-odds when all input features are zero.
• Maximum Likelihood Estimation (MLE):
  A method used to find the best values of weights and bias by maximizing the probability of observing the given data.

How to Evaluate Logistic Regression Model?

Evaluating the logistic regression model helps assess its performance and ensure it generalizes well to new, unseen data. The following metrics are commonly used:

1. Accuracy: Accuracy provides the proportion of correctly classified instances.

2. Precision: Precision focuses on the accuracy of positive predictions.

3. Recall (Sensitivity or True Positive Rate): Recall measures the proportion of correctly predicted positive instances among all actual positive instances.

4. F1 Score: F1 score is the harmonic mean of precision and recall.

5. Area Under the Receiver Operating Characteristic Curve (AUC-ROC)

• What it is: The ROC curve shows how well a model distinguishes between classes by plotting True Positive Rate (TPR) against False Positive Rate (FPR) at different thresholds. The AUC-ROC is the area under this curve.
• Interpretation:

— AUC = 1 → Perfect model

— AUC = 0.5 → Model is no better than random guessing

Advantages:

• Summarizes model performance across all classification thresholds.
• Useful for comparing different models.
• Not sensitive to class imbalance as much as accuracy.

Limitations:

• Can be misleading if classes are highly imbalanced.
• Doesn’t show the actual probability outputs or how well the model predicts positive cases specifically.

6. Area Under the Precision-Recall Curve (AUC-PR)

• What it is: The Precision-Recall (PR) curve plots Precision against Recall at different thresholds. AUC-PR is the area under this curve.

Interpretation:

• Higher AUC-PR → Model is better at correctly identifying positive cases.

Advantages:

• More informative than ROC when dealing with imbalanced datasets.
• Focuses on performance for the positive (minority) class.

Limitations:

• Can be less intuitive to interpret than ROC.
• Doesn’t account for true negatives.

Pros :

1. Probabilistic Output: Gives probability estimates for class membership (0–1).
2. Simple & Easy to Implement: Computationally efficient and widely used.
3. Interpretable: Coefficients show the effect of features on the outcome.
4. Handles Multiple Features: Can include many independent variables.
5. Works Well for Linearly Separable Data: Performs effectively when classes are linearly separable.
6. Less Prone to Overfitting: Compared to complex models, if features are few.

Cons :

1. Assumes Linear Relationship: Between features and log-odds of the outcome.
2. Sensitive to Outliers: Extreme values can distort results.
3. Limited to Linear Decision Boundaries: Cannot capture complex patterns without feature engineering.
4. Not Ideal for Small Datasets with Many Features: Can lead to unstable estimates.
5. Basic Logistic Regression is Binary: Extensions are needed for multi-class problems.