Logistic Regression in Machine Learning: Beyond Binary Classification and Straight Decision Boundaries

Image generated using AI tool

In my previous article on Logistic Regression, we explored the foundations of the algorithm: how a linear combination of features is converted into probabilities using the sigmoid function, how those probabilities are used for binary classification, and how Logistic Regression creates a decision boundary between two classes.

But binary classification is only one part of the story.

What happens when we have more than two classes?

And what happens when the relationship between our features and the target cannot be separated by a simple straight line? This is where Logistic Regression becomes more interesting.

In this second part, we will explore:

• Multiclass classification using Softmax Regression
• The difference between One-vs-Rest and Multinomial classification
• How Logistic Regression can learn non-linear decision boundaries
• Polynomial Logistic Regression
• The role of regularisation and hyperparameters
• Important Logistic Regression parameters in Scikit-learn

Let’s get into it.

From Binary Classification to Multiclass Classification

In binary Logistic Regression, we usually have two possible outcomes.

For example:

• Spam or not spam
• Fraud or not fraud
• Customer churns or does not churn
• Disease present or absent

The sigmoid function converts the output of our linear equation into a probability between 0 and 1:

where:

But now imagine we are trying to classify flowers into three categories:

• Setosa
• Versicolor
• Virginica

A single sigmoid output is no longer enough.

Instead of asking:

What is the probability that this observation belongs to class 1?

We now need to ask:

What is the probability of this observation belonging to each possible class?

And those probabilities should collectively add up to 1. This is where Softmax comes in.

Softmax Regression: Logistic Regression for Multiple Classes

Softmax Regression is essentially an extension of Logistic Regression for multiclass classification. Instead of producing a single probability, the model produces a probability distribution across all possible classes.

Suppose we have three classes. For a particular observation, the model might output:

• Class A: 0.05
• Class B: 0.25
• Class C: 0.70

The probabilities add up to. I mean obviously.

The model would therefore predict Class C.

The Softmax function converts the raw model outputs, often called scores or logits, into probabilities.

For class ii:

Here:

• z_i is the score for class ii
• K is the total number of classes
• e^{z_i} ensures that all values are positive
• Dividing by the total ensures that all probabilities sum to 1

The important idea is that Softmax does not look at one class in isolation.

The probability assigned to one class depends on the scores assigned to all the other classes as well.

A Simple Example Using the Iris Dataset

The Iris dataset is one of the most commonly used datasets for understanding multiclass classification.

It contains three flower species.

Let’s build a multiclass Logistic Regression model.

import seaborn as sns
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

We can load the dataset:

df = sns.load_dataset('iris')

encoder = LabelEncoder()
df['species'] = encoder.fit_transform(df['species'])

For simplicity, let’s use two features:

df = df[['sepal_length', 'petal_length', 'species']]

X = df.iloc[:, 0:2]
y = df.iloc[:, -1]

Now we split the dataset:

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=2
)

And train a multinomial Logistic Regression model:

clf = LogisticRegression(
    multi_class='multinomial'
)
clf.fit(X_train, y_train)

We can then make predictions:

y_pred = clf.predict(X_test)
print(accuracy_score(y_test, y_pred))

More importantly, we can inspect the probabilities produced by the model:

query = np.array([[3.4, 2.7]])
clf.predict_proba(query)

The output will contain one probability for each class.

For example:

[[0.02, 0.15, 0.83]]

The model is essentially saying:

“Based on these features, I believe there is an 83% probability that this observation belongs to the third class.”

We can then get the final predicted class using:

clf.predict(query)

This distinction between predict_proba() and predict() is worth remembering.

• predict_proba() gives us the probability distribution.
• predict() gives us the class with the highest probability.

One-vs-Rest vs Multinomial Logistic Regression

There are two major approaches for handling multiclass classification with Logistic Regression.

1. One-vs-Rest

Suppose we have three classes:

• A
• B
• C

One-vs-Rest creates three separate binary classification problems:

Model 1

A vs Not A

Model 2

B vs Not B

Model 3

C vs Not C

Each classifier independently calculates how likely an observation is to belong to its particular class. The final prediction is usually based on whichever classifier produces the strongest score. nThe main advantage of One-vs-Rest is simplicity.

However, the models are trained independently. They do not directly model the classes as one combined probability distribution.

2. Multinomial Logistic Regression

Multinomial Logistic Regression treats the classification problem as a single multiclass problem. Instead of building separate binary classifiers, it learns all classes together and uses Softmax to generate a probability distribution.

For mutually exclusive classes, this is often a more natural approach. For example, an Iris flower cannot simultaneously belong to Setosa and Virginica. The model needs to decide how probability should be distributed across all possible categories.

This is exactly what Softmax is designed to do.

In Scikit-learn, this can be specified using:

clf = LogisticRegression(
    multi_class='multinomial'
)

One important thing to remember is that the exact behaviour can also depend on the solver being used. And that brings us to an important aspect of using Logistic Regression in practice: hyperparameters.

But before that, let’s look at another limitation of standard Logistic Regression.

The Problem With Straight Decision Boundaries

Logistic Regression is called a linear model because its decision boundary is based on a linear combination of the input features.

With two features, the decision boundary may look like a straight line. With more features, it becomes a hyperplane. This works well when the classes can reasonably be separated using a linear boundary. But real-world data is not always that cooperative. Imagine a dataset where one class forms a curved pattern around another. A straight line would struggle to separate them.

This can lead to underfitting.

Does that mean we have to abandon Logistic Regression and immediately use a more complex model? Not necessarily. We can transform our input features.

Polynomial Logistic Regression

Polynomial Logistic Regression uses Polynomial Features to create additional features from the original dataset.

Suppose we originally have two features: X1 and X2

A polynomial transformation of degree 2 can generate features such as:

So instead of training the model only on:

X1, X2

we can train it on:

X1, X2, X1², X2², X1X2

The Logistic Regression model itself is still linear with respect to its parameters.

But because we have transformed the input space, the resulting decision boundary can become non-linear in the original feature space.

This is one of the most interesting ideas here.

We have not changed Logistic Regression into a non-linear algorithm. We have changed the representation of the data.

Creating Polynomial Features

Let’s start with a dataset containing two features and a binary target.

X = df.iloc[:, 0:2].values
y = df.iloc[:, -1].values

We can first train a normal Logistic Regression model:

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression()
clf.fit(X, y)

To evaluate it:

from sklearn.model_selection import cross_val_score

np.mean(
    cross_val_score(
        clf,
        X,
        y,
        scoring='accuracy',
        cv=10
    )
)

Now let’s transform the features.

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(
    degree=3,
    include_bias=False
)
X_trf = poly.fit_transform(X)

Now X_trf contains the original features along with their polynomial combinations.

We can train Logistic Regression again:

clf = LogisticRegression()

np.mean(
    cross_val_score(
        clf,
        X_trf,
        y,
        scoring='accuracy',
        cv=10
    )
)

If the underlying relationship in the dataset is non-linear, the polynomial transformation may significantly improve the model’s performance.

What Does the Polynomial Degree Actually Do?

The degree parameter controls how complex the transformed feature space becomes.

Get Priyanshu’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For example:

PolynomialFeatures(degree=2)

creates second-degree combinations.

Increasing the degree:

PolynomialFeatures(degree=5)

creates many more combinations and allows the model to learn more complex decision boundaries.

This sounds great. But there is a catch.

The Bias-Variance Trade-Off Appears Again

Suppose we experiment with different polynomial degrees.

for degree in range(1, 8):
    plot_decision_boundary(X, y, degree)

At a low degree, the decision boundary may be too simple. The model may fail to capture the actual pattern. This is underfitting.

As we increase the degree, the model gains flexibility. The decision boundary can bend and curve around the data.

At some point, we may achieve a good balance between:

• Capturing the underlying pattern
• Generalising to unseen data

But if we keep increasing the degree, the model can become excessively complex. A very high-degree polynomial can start fitting individual observations and noise. This is overfitting.

For example:

plot_decision_boundary(X, y, degree=25)

The resulting decision boundary may become extremely irregular.

It may perform very well on the training data while generalising poorly to new observations.

So the goal is not:

Use the highest polynomial degree possible.

The goal is:

Find the level of complexity that captures the underlying pattern without memorising noise.

This is why techniques such as cross-validation are extremely useful.

A Function to Experiment With Decision Boundaries

One useful way to understand Polynomial Logistic Regression is to train multiple models with different degrees and visualise their decision boundaries.

The basic workflow looks like this:

def plot_decision_boundary(X, y, degree=1):
  poly = PolynomialFeatures(degree=degree)
    X_trf = poly.fit_transform(X)
    clf = LogisticRegression()
    clf.fit(X_trf, y)
    accuracy = np.mean(
        cross_val_score(
            clf,
            X_trf,
            y,
            scoring='accuracy',
            cv=10
        )
    )

The function can then create a mesh of points across the feature space, transform those points using the same polynomial transformer, and ask the model to predict their classes.

This allows us to visualise how the decision boundary changes as the degree increases.

For example:

plot_decision_boundary(X, y, degree=1)

might produce a relatively simple boundary.

Then:

plot_decision_boundary(X, y, degree=3)

can capture more curvature.

And eventually:

plot_decision_boundary(X, y, degree=25)

may create an unnecessarily complicated boundary.

This visual experimentation makes the bias-variance trade-off much easier to understand than simply looking at accuracy scores.

Regularisation Becomes Even More Important

Polynomial transformations can dramatically increase the number of features.

More features mean more flexibility. But more flexibility also increases the risk of overfitting. This is where regularisation becomes important.

Logistic Regression in Scikit-learn allows us to control regularisation using parameters such as:

• penalty
• C
• l1_ratio

The penalty parameter can include regularisation methods such as:

penalty='l1'
penalty='l2'

or Elastic Net, depending on solver compatibility.

Understanding the C Parameter

One of the most important Logistic Regression hyperparameters is C.

This is easy to misunderstand.

C is inversely related to regularisation strength.

So:

Smaller C

C=0.1

means stronger regularisation.

Larger C

C=100

means weaker regularisation.

In simple terms:

• Small C → more restriction on model complexity
• Large C → model is allowed more flexibility

When using high-degree polynomial features, this becomes especially important.

If our polynomial expansion creates a large number of features, stronger regularisation can help prevent the model from assigning excessively large importance to individual features.

Choosing the Right Solver

Scikit-learn’s LogisticRegression has several solvers.

Some commonly encountered ones include:

• lbfgs
• liblinear
• newton-cg
• sag
• saga

The choice of solver matters because not every solver supports every combination of:

• Penalties
• Multiclass strategies
• Dataset sizes

For example, liblinear is useful for smaller datasets and supports L1 and L2 regularisation, but it handles multiclass classification through One-vs-Rest rather than true multinomial optimisation.

On the other hand, solvers such as lbfgs, newton-cg, sag, and saga can support multinomial classification.

saga is particularly flexible because it supports:

• L1
• L2
• Elastic Net

A typical configuration might look like:

clf = LogisticRegression(
    solver='saga',
    penalty='elasticnet',
    l1_ratio=0.5,
    max_iter=1000
)

Of course, this does not mean we should blindly use saga everywhere.

The right solver depends on the dataset, the regularisation method, and the problem we are trying to solve.

Other Important Hyperparameters

Let’s quickly look at a few more parameters that frequently matter in practice.

max_iter

max_iter=1000

This controls the maximum number of iterations the solver is allowed to use while trying to converge.

Sometimes you may see a warning telling you that Logistic Regression failed to converge.

Increasing max_iter may help, although convergence problems can also indicate issues with feature scaling or the data itself.

class_weight

For imbalanced datasets, we may use:

class_weight='balanced'

This automatically adjusts the importance assigned to different classes based on their frequencies.

This can be useful when one class is significantly underrepresented.

tol

The tol parameter controls the tolerance for optimisation.

In simple terms, it helps determine when the solver should stop because further improvements have become sufficiently small.

warm_start

When:

warm_start=True

the model can reuse the solution from a previous fit as the starting point for the next optimisation.

This can be useful in certain iterative experimentation workflows.

Putting Everything Together

At this point, we can see that Logistic Regression is much more flexible than it may initially appear.

For binary classification, we use the sigmoid function.

For multiclass classification, we can use Softmax to generate probabilities across multiple classes.

When a straight decision boundary is insufficient, we can use Polynomial Features to transform the input space and allow Logistic Regression to learn curved boundaries.

And when the model becomes too flexible, we can control complexity using regularisation.

The general workflow might look something like this:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LogisticRegression

model = Pipeline([
    ('poly', PolynomialFeatures(degree=3)),
    ('logreg', LogisticRegression(
        C=1.0,
        max_iter=1000
    ))
])
model.fit(X_train, y_train)

Using a pipeline is especially useful because it ensures that the same feature transformation is applied consistently during training and prediction.

Final Thoughts

Before studying Logistic Regression in more detail, I used to think of it as a relatively simple algorithm mainly used for binary classification.

But there is much more happening beneath the surface.

Logistic Regression can:

• Handle binary classification using the sigmoid function
• Extend to multiple classes using Softmax
• Use different multiclass strategies such as One-vs-Rest and Multinomial classification
• Learn curved decision boundaries through polynomial feature transformation
• Control complexity using regularisation
• Be tuned through multiple hyperparameters and solvers

The most interesting takeaway for me from this topic is probably the idea behind Polynomial Logistic Regression.

The model itself remains linear with respect to its parameters. But by changing the feature space, we can create surprisingly complex decision boundaries.

That idea goes beyond Logistic Regression.

A lot of machine learning is not just about choosing a more powerful model. Sometimes, how we represent the data can completely change what a relatively simple model is capable of learning.

And that is something I am slowly beginning to appreciate more as I continue rebuilding my understanding of Machine Learning from the ground up.

On to the next topic. 🚀