Linear Classifiers

23 October, 2025

1 Introduction: What is a linear classifier?

A linear classifier is one of the simplest machine learning models. It separates linearly separable data into two classes using a linear decision boundary. In a two-dimensional feature space, this boundary is a line, in three dimensions it becomes a plane, and in n dimensions it generalises to an (n − 1) dimensional hyperplane.

Imagine we want to separate apples and bananas based on two features: redness (low for bananas, high for apples) and their roundness (low for bananas, high for apples). We could plot each fruit’s features as a point on a 2D graph (redness on the x axis, aspect ratio on the y axis).

The linear classifier model would then find a decision boundary that would classify points on one side of it as apples and points on the other side as bananas. In general, a linear classifier finds the ‘best’ hyperplane that splits one class from the other in the training data, to separate new unseen examples correctly.

Figure 1: Apple and banana datapoints visualised on a graph.

2 The Decision Boundary

Mathematically, the decision boundary is a n−1 plane existing in a n-dimensional feature space, for our case, the two features are redness and roundness. Thus, our hyperplane is given by the equation:

w1 · x1 + w2 · x2 + b = 0

Which is the same idea as the equation below, but with different variable names: ax + by + c = 0

Here, x1 and x2 are our features — redness and roundness — in place of the variables x and y, and w1 and w2 are the weights, in place of a and b — coefficients that control the slope of the hyperplane. The bias b (or c in the other representation) controls the vertical offset from the origin. Finally, we define a function f(x) that encases our equation from before:

f(x) = w1 · x1 + w2 · x2 + b

Get mogthecat’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

When f(x) = 0, we get the decision boundary line. From that, we further define a characteristic function ŷ to allow the model to make predictions:

Thus, the hyperplane f(x) = 0 splits our feature space into two sections, which are then labelled ‘apple’ and the other ‘banana’ by the function yˆ. We often assign +1 to the positive side ‘apple’ and 0 to the negative side ‘banana’. What we have done so far is essentially assigning +1 and 0 values to input datapoints multiplied by some weights, which are ‘learnt’ (explained in the next section).

Visualisation: As apples are typically red (high redness) and relatively round (high roundness), and bananas are yellow (low redness) and long (low roundness), a good model might have a large positive weight for both redness (w1) and roundness (w2). The bias b sets a threshold for when f(x) turns positive. For a new fruit, if f(x) > 0, we predict apple; and if f(x) < 0, we predict banana.

Figure 2: Datapoints separated by a linear decision boundary.

3 The Loss Function: The Error of the Model

To find a suitable w1, w2, and b, we first have to define a loss function that quantifiably measures the mistakes or errors the model is making. In general, loss functions calculate the differences between the model’s predictions (yˆ) and the true values (y). For our case, yˆ and y can only be one of two values: 0 for banana and 1 for apple. A common loss function is Mean Squared Error:

Remember: A loss function tells us how much ‘error’ a model is making, or its mistakes. If we can minimise the loss, it would mean the model’s outputs are getting closer to the expected and correct values, which would mean fewer mistakes and a higher accuracy.

Figure 3: An example loss function is shown, taking the weights x and y and outputting a value z, which is the loss at that ‘point’.

Here, x is in place of w1 and y is for w2. The output is z, a scalar value telling us the loss at a specific point. If we want to minimise ‘z’ (which again tells us the error with some specific weights here visualised as the height of the graph), we need to differentiate the loss function and find the gradient using a technique called backpropagation. This can then be numerically implemented using optimisation algorithms such as gradient descent, RMSprop, Adagrad, and Adam, which in a short sense are different ways of updating the weights of the model.

4 Conclusion

Linear classifiers separate data with a linear decision boundary, making decisions by passing data points through the function f. When f(x) > 0, we sorted it to one class, and the other when f(x) < 0. Improving a model entails choosing a loss function and minimising it using an optimiser algorithm such as gradient descent.

Other aspects affect a model’s performance that were not covered such as under/overfitting and techniques like L2 regularisation that penalise large weights. Despite their relative simplicity, linear classifiers are foundational to more complex neural network architectures such as support vector machines, perceptrons, and the modern-day transformer architecture.