Can we use linear regression for classification tasks?
The simple answer is Yes but not suitable. I know you just learned how to predict housing prices using linear regression now you want to see If you can use it to predict whether the patients will be diagnosed with cancer or not or if you want to classify the good vs bad tweets then I have bad news for you, linear regression is not well equipped to do these classification tasks. The first well-known reason is that linear regression deals with linearly distributed data not with data clustered in discrete groups and the second reason is outliers, if you got your answer you don’t need to read further. But if you want to know about decision boundary, how to convert continuous values to discrete values, and how the linear regression reacts to outliers then you are most welcome to read further.
Decision Boundary just by looking at the dataset
The first data set asks you to predict either 1 or 0. We clearly know that linear regression predicts continuous values and not just 1s and 0s. So our first job gonna be to transform the continuous values into discrete values. But wait, do we even need linear regression to solve this problem? What if we just draw a boundary at around radius=14 and define a simple function such that if tumour_size > 14, classify into categories 1 else classify into categories 0, what do you think? just see this.
According to this model, all the data on the right of the green dotted line are diagnosed as having malignant tumors and benign tumors on the left. I know there are a few misclassifications. The accuracy of this model is 0.85. That means out of 100 labeled data 85 are correctly classified and 15 are wrongly classified. By the way, the green dotted boundary is also called a decision boundary in machine learning lingo. This decision boundary was decided arbitrarily just by taking a glance at the dataset. Do you think linear regression will help us to find a better decision boundary? Just scroll down to find the answer.
Decision Boundary decided by Linear regression
Let’s first fit a line and using the parameters of the fitted line we can easily find the decision boundary. But first, we should know where should we divide the line. I mean to say that linear regression outputs values ranging from -infinity to +infinity, then how should we know which outputs to classify as 1s and which to 0s? Looking at the dataset there is no doubt that we can split the predicted results at 0.5. All the predicted results greater than 0.5 will be classified as 1s and 0s otherwise. To find the decision boundary we have to know for what values of tumour_size(x) the diagnosis(y) value is 0.5, We trained linear regression and calculated our linear model coefficient(θ1=0.1), and intercept(θ0=-1.04), and we know y=0.5, then find the value of x if (θ1X +θ0=Y). Isn’t like solving a linear equation? Well, I got x=15.4, It means according to linear regression we should put our decision boundary at tumour_size 15.4, not at 14. Even the accuracy of this linear model is just 0.87. So far so good, we tweaked a little bit here and there and made a linear regression to solve a classification problem. Seems like it worked, isn’t it?
What if there are outliers?
That lies an abnormal distance from other values, will the accuracy of the linear model remain the same? Look below, just because of two data points at tumor sizes 150 and 155 the classification accuracy of the linear model dropped to 0.66. More than half of the Malignant (1) is misclassified as benign(0).
Now what?
Well, linear regression tried its best to do its job but for classification, we need a formula or function that can squeeze the outputs in the range between (0,1) and the fitting line should be a squiggly line like a z shape instead of a straight line so that the model won’t get altered by the outliers. See this, someone already has found this function called a logistic function. If you are in hurry to know about it then go along with the links below, if not then wait for my tutorial on logistic regression. Until then stay safe stay happy, and thanks for being with me until here. God Bless you.
I just want to leave you with a question, Is accuracy a good metric for classification tasks? My model works perfectly 99% of the time, just in one case, it failed. well, Just one positive covid case is sufficient to cause havoc.
References
This Tutorial: https://www.kaggle.com/code/ravin235/linear-to-classification
Get Ravindra Sah’s stories in your inbox
Join Medium for free to get updates from this writer.
Linear Regression:https://medium.com/swlh/linear-regression-for-beginners-with-implementation-in-python-3f99fe470ef3
Linear Regression not suitable for classification: https://youtu.be/2TvKZnTHC4M
Logistic Regression Visualization:https://www.kaggle.com/code/aakritsinghal/logistic-regression-visualisation
Andrew Ng’s Machine Learning Course in Python (Logistic Regression):https://towardsdatascience.com/andrew-ngs-machine-learning-course-in-python-logistic-regression-c0ae25509feb
