Multiple-Binary Class Classification

Supervised Classification models are used to categorize data into predefined classes or labels. They learn patterns from the labeled training data to predict the class of new, unseen instances. Logistic Regression, Support Vector Machines, Naive Bayes, Decision Trees, K-NN are among them.

Some of these models such as Logistic Regression, SVM, LDA inherently supports binary class classification. And to make these algorithms multi-class classification models, special strategies are used which breaks the multi-class into multiple binary classes. They are:

• OVA(One-Verses-All) or OVR(One-Verses-Rest) and
• OVO(One-Verses-One).

One-Verses-All(OVA) or One-Verses-Rest(OVR):

The OVA/OVR involves creating n binary class classifiers for n classes, resulting in training one binary classifier per class. Each classifier is trained to distinguish the class of interest as positive(1) and the others as negative(0), based on a output score, such as probability. The classifier, among all the n classifiers, that gives highest score wins and it is the final prediction.

Example: For 3 class classification(C1,C2,C3), 3 classifiers will be built as below:

Classifier1: predicts whether the inputs belongs to C1 or not i.e. C1 vs {C2,C3}

Classifier2: predicts whether the inputs belongs to C2 or not i.e. C2 vs {C1,C3}

Classifier3: predicts whether the inputs belongs to C3 or not i.e. C3 vs {C1,C2}

The classifier that results positive output or the one with highest score among the classifiers is the final prediction.

One vs All Strategy using 3 class problem

Pros:

• OVA is relatively easy to implement as it takes the advantage of binary classifiers.
• Works well when the dataset is large and class overlap isn’t significant(i.e. least overlap among the classes)
• As the classifier handles one vs rest, the training process can be done independently, making the process easy to manage in terms of complexity.
• Each classifier provide clear decision boundary for a single class, making it easier to understand and interpret

Cons:

• A major challenge in OVA is imbalance in class distribution. Each classifier will be trained with the dataset to distinguish one class from the rest, which sometimes may result in an imbalance between positive class( the class of interest) and the negative class(all other classes).
• As the classifier work independently, the confidence scores they produce may not always be comparable across classes, making the final decision process ambiguous, especially if the scores are too close.
• There is a risk of overfitting as the model gets trained multiple times on multiple classes.
• It may also struggle with the overlapping classes, as it requires each classifier to make broad distinctions between one class and all others.

One-Verses-One:

Get Ginjupalli Yegnesh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

As the name suggests, the strategy involves training a binary classifier for every possible pair of classes. For n classes, this strategy involves creating n(n-1)/2 classifiers and each classifier trained to distinguish between two classes.

During classification, each classifier casts a vote for one of the two classes it was trained on. The class with the most votes is the one that is selected as the final prediction.

Example: For 3 classes, 3(3–2)/2 classifiers will be built:

Classifier 1: Predicts between C1 and C2

Classifier 2: Predicts between C1 and C3

Classifier 3: Predicts between C2 and C3

During prediction, all classifiers are applied, and each classifier will vote to one of the classes it was trained on. The class with highest number of votes is chosen as the final prediction.

One vs One strategy using 3 class problem

Pros:

• OVO produces more refined decision boundaries as each classifier only has to distinguish between two classes. This is suitable when the class overlap is significant.
• As each classifier deals with a smaller subset of data(only two classes at a time), OVO is less affected by class imbalances compared to OVA/OVR.
• OVO is well suited for algorithms like SVM, which are computationally expensive but excel in binary classification tasks.

Cons:

• Training n(n-1)/2 classifiers can be computationally expensive, especially for datasets with many classes.
• Prediction time can be slower as it requires voting among all classifiers, which can be significant if there are many classes.

OVA/OVR vs OVO:

Comparison of OVA vs OVO

References:

S. Andhalkar and B. F. Momin, “Multiclass IFROWNN Classification Algorithm Using OVA and OVO Strategy,” 2018 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Bengaluru, India, 2018, pp. 1–7, doi: 10.1109/ICCCNT.2018.8493865.