Understanding Logistic Regression

Many real-world machine learning problems require predicting categories rather than continuous values. For example, we can determine whether an email is spam, whether a customer will purchase a product, or whether a medical test is positive or negative. Such problems are known as classification problems.

Although Linear Regression can model relationships between variables, it is not well-suited for classification because its predictions are unbounded and cannot be interpreted as probabilities. Logistic Regression addresses this limitation by estimating the probability that an input belongs to a particular class. It combines a linear model with the sigmoid function, which maps any real-valued input to a value between 0 and 1, making it suitable for binary classification.

Index

1. What is Logistic Regression?
2. Why Sigmoid?
3. Logistic Regression Workflow
4. Odds, Log-Odds, and the Logistic Function
5. Maximum Likelihood Estimation (MLE)
6. Optimising Logistic Regression
7. Confusion Matrix
8. Advantages and Limitations of Logistic Regression
9. Bonus: Logistic Regression as the Building Block of Neural Networks
10. Conclusion

1. What is Logistic Regression?

Logistic Regression is a supervised machine learning algorithm used for classification problems, where the objective is to predict the probability that an input belongs to a particular class. Unlike Linear Regression, which predicts continuous numerical values, Logistic Regression predicts the probability of an observation belonging to one of the predefined classes. It is primarily used for binary classification, where the target variable has only two possible outcomes.

A natural question is why we don’t simply use Linear Regression for this task. If we fit a straight line to a 0/1 target, two problems arise. First, Linear Regression can output any value from −∞ to +∞, but a class prediction should be bounded; a probability only makes sense between 0 and 1, and a raw linear output has no such guarantee. A model might predict 1.4 or −0.3 for a given input, and neither value has any meaningful interpretation as “belonging to a class.” Second, Linear Regression assumes a linear relationship between the input and the target and normally distributed errors, but a binary target can only take the values 0 or 1, so these assumptions do not hold. The model also becomes sensitive to outliers, which can pull the fitted line — and with it, the decision boundary — in misleading directions.

Logistic Regression solves this by keeping a linear combination of the inputs on the inside, but wrapping it in a function — the sigmoid — that squashes the result into the (0,1) range so that the output can be read directly as a probability.

Consider the problem of classifying an email as spam or not spam. Instead of directly predicting the class label, Logistic Regression estimates the probability that the email belongs to the spam class. Mathematically, it models

where Y is the target variable, X represents the input features, and P(Y=1∣X=x) denotes the probability that the input x belongs to class 1. Like Linear Regression, the model first computes a linear combination of the input features,

where θ is the vector of model parameters (weights), and b is the bias (intercept). Since z can take any real value from −∞ to +∞, it cannot be interpreted as a probability. To overcome this limitation, Logistic Regression passes z through the sigmoid (logistic) function, which maps every real-valued input to the interval (0,1):

The predicted probability is then converted into a class label by comparing it with a decision threshold t, where 0≤t≤1.

The choice of the threshold depends on the application. While a threshold of 0.5 is commonly used in practice, other values may be chosen to balance false positives and false negatives.

Although its name contains the word “Regression,” Logistic Regression is fundamentally a classification algorithm. The term regression arises because the model assumes a linear relationship between the input features and the log-odds of the probability, rather than between the features and the target variable itself.

2. Why Sigmoid?

In Logistic Regression, the linear model computes

where z can take any real value from −∞ to +∞, however, the objective of Logistic Regression is to estimate the probability that an observation belongs to a particular class.

Since probabilities must always lie between 0 and 1, the linear output cannot be used directly. Therefore, a transformation is required to convert any real-valued input into a valid probability.

An ideal probability function should satisfy the following properties:

• Bounded Output: The output should always lie between 0 and 1 so that it can be interpreted as a probability.
• Monotonicity: Larger values of the linear score should correspond to higher probabilities, while smaller values should correspond to lower probabilities.
• Continuity: Small changes in the input should produce small changes in the output, ensuring stable predictions.
• Differentiability: The function should be differentiable so that gradient-based optimisation algorithms can efficiently learn the model parameters.
• Computational Simplicity: The function should be mathematically simple enough to allow efficient optimisation.

The sigmoid function satisfies all these requirements and is therefore used in Logistic Regression. It is defined as

The sigmoid function maps every real number to the interval.

making its output directly interpretable as a probability.

Because of these properties, the sigmoid function provides a smooth transition between the two classes instead of making abrupt decisions. This smooth behaviour also allows the model parameters to be learned efficiently using gradient-based optimisation techniques.

Although other activation functions, such as ReLU and tanh, are widely used in machine learning, they are not suitable for Logistic Regression. ReLU is unbounded above and therefore cannot represent probabilities, while tanh produces outputs in the interval (−1,1), which includes negative values that cannot be interpreted as probabilities. Consequently, the sigmoid function remains the most appropriate choice for binary classification.

For multiclass classification, where an observation belongs to one of K mutually exclusive classes, the Softmax function is used instead of the sigmoid function. While the sigmoid function estimates the probability of a single class, the Softmax function converts the output scores across all classes into a valid probability distribution in which the probabilities sum to 1. For a model producing scores z1,z2,…,zK, the probability of the ith class is given by

where

This extension of Logistic Regression, known as Multinomial Logistic Regression or Softmax Regression, is widely used for multiclass classification problems.

3. Logistic Regression Workflow

4. Odds, Log-Odds, and the Logistic Function

In the previous sections, we saw that Logistic Regression starts with a linear score and transforms it into a probability using the sigmoid function. But why does the sigmoid function have exactly this form?

To establish this mapping, we first introduce the concepts of odds and log-odds.

Odds

Suppose an event has probability p of occurring.

The probability that it does not occur is

The odds of the event are defined as the ratio between these two probabilities:

Log-Odds (Logit)

Although odds eliminate the upper bound imposed on probabilities, they remain restricted to positive values and therefore cannot represent the full range of the linear model. To obtain a quantity whose range matches that of the linear model, we take the natural logarithm of the odds:

This quantity is also called the logit.

Since the logarithm maps the interval (0,∞) to (−∞,∞), the log-odds can take any real value:

This is precisely the same range as the linear model.

Therefore, we equate the log-odds to the linear model:

where

• x is the input feature vector,
• θ is the vector of model weights,
• b is the bias,
• p is the probability of class 1.

From log odds to sigmoid

take the exponential of both sides:

This is exactly the sigmoid function applied to the linear score.

We define

or equivalently,

This tells the probability that the input sample belongs to class 1. This is our logistic model.

We can find the probability that the input sample belongs to class 0 as:

Since the target variable in binary classification has only two possible outcomes (0 or 1), it can be modelled as a Bernoulli random variable. Just as a coin toss results in either heads or tails, each observation in binary classification belongs to exactly one of two classes. Therefore, the Bernoulli distribution is the natural choice for modelling the target variable in Logistic Regression. The only difference is that, unlike a fair coin whose probability remains fixed, Logistic Regression estimates the probability of each outcome from the input features.

For multiclass classification, where the target variable can take more than two mutually exclusive outcomes, the Bernoulli distribution is replaced by the Multinoulli (Categorical) distribution, which models the probability of each possible class.

5. Maximum Likelihood Estimation (MLE)

In the previous section, we modelled the target variable using the Bernoulli distribution. For a given input xi​, the probability of observing the corresponding class label yi​ is given by:

where

and

Since yi can only take the values 0 or 1, there are only two possible cases.

Case 1: yi=1:

Case 2: yi=0

the Bernoulli distribution combines both into a single compact expression

This compact representation is possible only for binary classification, where the target variable takes values.

our model now describes the probability of observing a single training example. However, during training, we are given an entire dataset rather than a single observation.

Get Ujjjwalx’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Since the observations are independent, the probability of observing the entire dataset is the product of the probabilities of the individual training examples:

Substituting the Bernoulli probability model,

This function is called the likelihood function. It measures how likely the observed training data are for a given choice of model parameters.

The central idea of Maximum Likelihood Estimation (MLE) is to determine the parameter values that maximise this likelihood.

However, directly maximising the likelihood is inconvenient because it is the product of many probabilities, which can become extremely small, leading to numerical underflow. Therefore, instead of maximising the likelihood itself, we maximise its logarithm, called the log-likelihood, which yields the same optimal parameters while making the optimisation much simpler.

Substituting the likelihood function,

Finally, substituting

we obtain the log-likelihood function for Logistic Regression:

The objective of Logistic Regression is therefore

Since most optimisation algorithms are designed to minimise an objective function rather than maximise it, we convert the maximisation problem into an equivalent minimisation problem by introducing a negative sign. Thus, instead of maximising the log-likelihood, we minimise the negative log-likelihood (NLL):

For convenience, the loss is usually averaged over all m training examples:

This function is known as the Binary Cross-Entropy (BCE) Loss, Log Loss, or Negative Log-Likelihood (NLL). The name Binary Cross-Entropy arises because the loss computes the average entropy of the true binary class with respect to the model's predicted probability distribution. Lower entropy indicates that the model assigns a higher probability to the correct class, while higher entropy indicates greater prediction error.

During training, the objective of Logistic Regression is to find the model parameters θ and b that minimise this loss.

This loss function forms the basis for optimising Logistic Regression using iterative optimisation algorithms such as Gradient Descent.

We can also use mean squared error loss instead of binary cross-entropy, but it is not the appropriate choice for binary classification. It does not arise from the Bernoulli Likelihood, and it produces smaller gradients when combined with the sigmoid function, resulting in slower convergence.

6. Optimising Logistic Regression

Unlike Linear Regression, Logistic Regression does not have a closed-form analytical solution for estimating the model parameters. This is because the cost function contains the nonlinear sigmoid function, making it impossible to directly solve for the optimal parameters by setting the derivatives equal to zero.

Consequently, Logistic Regression relies on iterative optimisation algorithms that gradually update the model parameters until the loss function reaches its minimum. Two classical approaches are the Perceptron Learning Algorithm and Gradient Descent.

Perceptron Learning Algorithm

The Perceptron Learning Algorithm (PLA) is one of the earliest algorithms proposed for binary classification. Like Logistic Regression, it attempts to learn a linear decision boundary by updating the model parameters whenever a training sample is misclassified.

For a training sample xi​, the predicted class is

If the prediction is incorrect, the parameters are updated according to

where

• η is the learning rate,
• yi​ is the true class label,
• yi is the predicted class label.

Working of the Perceptron Algorithm

1. Initialise the weights and bias.
2. Select a training sample.
3. Compute the predicted class.
4. If the sample is correctly classified, no update is performed.
5. If the sample is misclassified, update the weights and bias using the above equations.
6. Repeat the process until convergence or until a maximum number of iterations is reached.

Limitations

Although historically important, the Perceptron Learning Algorithm has several limitations:

• It converges only when the training data are perfectly linearly separable.
• If the classes overlap, the algorithm keeps updating the parameters indefinitely and may never converge.
• Since it uses only the predicted class labels rather than predicted probabilities, it ignores the confidence of the predictions.
• In practice, it often requires many iterations before reaching a solution.

Because of these limitations, the Perceptron algorithm has largely been replaced by optimisation methods based on differentiable loss functions.

Gradient Descent

The Gradient Descent algorithm used in Logistic Regression closely resembles the Perceptron Learning Algorithm. However, unlike the Perceptron, which updates its parameters using the predicted class label, Logistic Regression minimises the Binary Cross-Entropy (Negative Log-Likelihood) loss function derived earlier by updating its parameters using the predicted probability from the sigmoid function.

The objective function is

where

Gradient Descent updates the parameters in the direction opposite to the gradient of the loss function, since the gradient points toward the direction of maximum increase.

The parameter updates are

For Logistic Regression,

Thus, the update equations become

It is interesting to observe that the Gradient Descent update equations for Logistic Regression closely resemble those of the Perceptron Learning Algorithm. In both methods, the model parameters are updated based on the prediction error. The key difference lies in the quantity used to compute this error. The Perceptron updates its parameters based on the predicted class label, resulting in discrete updates. In contrast, Logistic Regression updates its parameters based on the sigmoid function's output, yielding smooth, continuous updates. Consequently, Logistic Regression not only determines whether a prediction is correct or incorrect but also considers the confidence of that prediction, leading to more stable optimisation and better performance on datasets that are not perfectly linearly separable.

7. Confusion Matrix

After training a Logistic Regression model, it is necessary to evaluate its performance on unseen data. While the Binary Cross-Entropy loss measures the error during training, it does not directly indicate how many predictions were correct or incorrect. For this purpose, a Confusion Matrix is commonly used.

A Confusion Matrix summarizes the performance of a binary classifier by comparing the actual class labels with the predicted class labels.

The four possible outcomes are:

• True Positive (TP): The model correctly predicts the positive class.
• True Negative (TN): The model correctly predicts the negative class.
• False Positive (FP): The model incorrectly predicts the positive class when the true class is negative (Type I Error).
• False Negative (FN): The model incorrectly predicts the negative class when the true class is positive (Type II Error).

Using these four quantities, several evaluation metrics can be computed.

Accuracy

Accuracy measures the proportion of correctly classified samples.

Although accuracy is simple to interpret, it can be misleading for highly imbalanced datasets.

Precision

Precision measures how many of the samples predicted as positive are actually positive.

A high precision indicates a low number of false positive predictions.

Recall (Sensitivity)

Recall measures how many of the actual positive samples are correctly identified.

A high recall indicates that the classifier misses very few positive samples.

Threshold controls how difficult it is to call something positive. If we are increasing threshold, we are making more difficult for prediction to be classified as Positive.

Increasing threshold decreases number of True Positive and False Positive. Since FP factor of precision has greater effect than decrease in TP factor therefore precision improves.

Similarily, it increases number of false negatives and true negatives. Since FN factor of recall increases, so recall decreases with increase in threshold

F1-Score

Since precision and recall often exhibit a trade-off, relying on only one of these metrics may provide an incomplete assessment of a classifier’s performance. Therefore, the F1-score is used as a single evaluation metric that balances both precision and recall by computing their harmonic mean.

The F1-score is particularly useful for evaluating classifiers on imbalanced datasets, where considering both false positives and false negatives is equally important.

8. Advantages and Limitations of Logistic Regression

Logistic Regression is one of the most widely used classification algorithms due to its simplicity, interpretability, and strong mathematical foundation. It is computationally efficient, easy to implement, and produces probabilistic outputs that can be converted into class labels using a suitable decision threshold. Since the model parameters have a clear interpretation, it is often preferred in applications where explainability is important, such as healthcare, finance, and risk analysis. Furthermore, the Binary Cross-Entropy loss is a convex function, allowing Gradient Descent to converge reliably to the global optimum. These properties also make Logistic Regression an excellent baseline model for many classification problems.

Despite these advantages, Logistic Regression has several limitations. The model assumes a linear decision boundary, making it unsuitable for problems with complex nonlinear relationships unless appropriate feature engineering is performed. Its performance is sensitive to the quality of the input features and can be affected by multicollinearity and outliers. In addition, although extensions such as Multinomial Logistic Regression (Softmax Regression) enable multiclass classification, the standard Logistic Regression model is inherently designed for binary classification. Consequently, for highly complex datasets, more expressive models such as decision trees, support vector machines, or neural networks often provide better predictive performance.

9. Bonus: Logistic Regression as the Building Block of Neural Networks

Although Logistic Regression is often studied as a standalone classification algorithm, it also serves as the fundamental building block of modern neural networks. In fact, a single neuron in a neural network performs almost the same computation as a Logistic Regression model.

For an input feature vector x, a neuron first computes a weighted sum of its inputs,

which is identical to the linear model used in Logistic Regression. This value is then passed through an activation function. If that activation function is the sigmoid,

the neuron outputs the probability that the input belongs to the positive class — exactly as in Logistic Regression. During training, the neuron is optimised using Binary Cross-Entropy (equivalently, Negative Log-Likelihood) and Gradient Descent, the same optimisation procedure derived earlier in this article.

The key difference lies in scale and structure. Logistic Regression consists of a single computational neuron with no hidden layers, so it can only learn a linear decision boundary. A neural network, by contrast, stacks many such neurons across one or more hidden layers, where each neuron learns a different feature representation of the input. Hidden layers typically use nonlinear activation functions such as ReLU, tanh, or GELU, which precisely allow the network to model complex, nonlinear relationships that a single logistic unit cannot. The output layer then uses whichever activation suits the task — sigmoid for binary classification, softmax for multiclass classification, or a linear activation for regression.

Seen this way, Logistic Regression is the simplest possible neural network: one neuron, no hidden layers, sigmoid output. Understanding it thoroughly — the linear combination, the activation, the loss function, and how gradients flow back to update the parameters — is what makes the mechanics of deeper networks feel like a natural extension rather than a new set of rules.

10. Conclusion

Logistic Regression is one of the most fundamental and widely used algorithms for binary classification. Unlike Linear Regression, which predicts continuous values, Logistic Regression models the probability that an observation belongs to a particular class by combining a linear model with the sigmoid function. This probabilistic formulation naturally leads to the Bernoulli distribution, Maximum Likelihood Estimation, the log-likelihood function, and ultimately the Binary Cross-Entropy loss used to train the model.

Although this optimisation problem has no closed-form analytical solution, iterative methods such as Gradient Descent efficiently learn the optimal parameters by minimising the Binary Cross-Entropy loss. Once trained, the model can be evaluated using the Confusion Matrix and metrics such as Accuracy, Precision, Recall, and F1-score, which together give a comprehensive picture of its performance.

Despite its simplicity, Logistic Regression remains an important algorithm — not just as a teaching tool, but as a genuinely useful one, valued for its computational efficiency, interpretability, and strong mathematical foundation. Perhaps more importantly, it forms the conceptual basis of modern neural networks: a single neuron with a sigmoid activation performs the same computation as a Logistic Regression model. Understanding Logistic Regression, then, is not just about mastering one algorithm — it’s about building the mathematical intuition that underlies deep learning and modern artificial intelligence as a whole.