Supervised machine learning algorithm — Logistic regression
While there are many machine learning algorithms for classification problems (i.e to predict whether the input is ‘cat’ or ‘dog’ , ‘car’ or ‘truck’ etc.,) logistic regression is one of the simple, easy to understand, powerful algorithm for classification.
Let’s see about how logistic regression works step by step and also the python coding involved in it.
For example, consider the following dataset. It provides information about whether a cancer cell is malignant(dangerous) or benign(not dangerous) based on the tumor’s size and cellular density. If we train logistic regression model with this data, the model should be able to provide predictions for unseen(new) data.
The steps involved in logistic regression are as follows
- Obtain the logistic regression function
- Calculate Loss (error) of the predictions
- Obtain cost function of the Loss (so that the loss can be minimized by finding optimum value of w and b)
- Form decision boundary for future predictions.
1. Obtain the logistic regression function
The following logistic function — a type of sigmoid function is used to predict the output in logistic regression.
We might have heard of the formula to create linear line y = mx+c. By replacing y with z, m with w, c with b, we get the following formula. If you wonder what these value mean , you can refer it to here.
By substituting, z in g(z) formula , we get
This is the logistic regression function that can help us perform logistic regression. The value of g(z) can be between 0 and 1. Therefore there requires necessity of setting threshold value.
If the threshold value is set to 0.5, all values less than 0.5 will be predicted as 0 and all values equal or above 0.5 is predicted as 1.
2. Calculate Loss (error) of the predictions
This is the step it gets complicated in comparision with linear regression. In case of linear regression, the loss can be calculated as difference between actual and predicted values.
However, for logistic regression
a) To calculate loss incase of y=1, -log(x) is used
b) To calculate loss incase of y=0, -log(1-x) is used
To understand the above, lets visualize these functions in graph.
a) loss incase of y=1, -log(x) is used to calculate error.
Lets plot graphs log(x) and -log(x). You can see that the both functions coincide at x =1.
Focusing only on green dotted section in the above graph, we obtain the following.
This function is used to calculate error in case of y=1. For example , in the above plot, in case the prediction is green dot (i.e value of x = prediction = 1)as shown and the actual value is 1, the error is 0 (i.e value of -log(x)) . In case the prediction is blue dot (i.e value of x = prediction =0.4 and if threshold is set to 0.5, the prediction = 0 ) and the actual value is 1, the error will be infinite ( i.e the value of -log(x)).
Get Santhosh Sivan’s stories in your inbox
Join Medium for free to get updates from this writer.
b) loss incase of y=0, -log(1-x) is used to calculate error.
Lets plot graph -log(1-x)
This function is used to calculate error in case of y=0. For example , in the above plot, in case the prediction is green dot (i.e value of x = prediction = 0)as shown and the actual value is 0, the error is 0 (i.e value of -log(1-x)) . In case the prediction is blue dot (i.e value of x = prediction =0.8 and if threshold is set to 0.5, the prediction = 1) and the actual value is 0, the error will be infinite ( i.e the value of -log(1-x)).
Therefore, the calculation of loss for 1 input can be done by represented by the following function.
3. Obtain cost function of the Loss
The loss function of all elements in a dataset can be written as follows.
Cost function means to find average of losses of the function. It can be written as follows.Note that cost function is used to find optimum values of weights (w) and bias (b) so that the loss is minimum.
The above function can be rewritten as follows. note that y_i is added so that in case of y=0 or y=1, their appropriate function only remained and the other gets cancelled out.
Next, this cost function J(w,b) is minimized using differential calculus to obtain best value for w and b.
4. Form decision boundary for future predictions.
By substituting the found w and b values , decision boundary for the above cancer data is formed as follows. This decision boundary is used to predict and classify new unseen datas in future.
Enough theory, let’s see code to perform the above steps.
we will be using this dataset to perform logistc regression. There are some data preprocessing steps needs to be done inorder to perform logistic regression . I will skip coding required for those steps. However you can take a look on my github if you are interested.
# All of the above steps will be performed by the following coding.
# Fitting the Logistic Regression into the Training set
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss
logr = LogisticRegression(random_state = 0)
logr.fit(X_train_encoded, y_train)
# make prediction
prediction = logr.predict(X_valid_encoded)
# print weights and bias of the model after minimizing cost function.
print(f"Intercept: {logr.intercept_}") # prints Intercept: [-1.41643911]
print(f"Coefficient: {logr.coef_[0]}") # prints Coefficient: [ 0.71839823 0.38765722 3.89528765 -0.00653087 0.00653087]
# calculate log loss of the model
log_error = log_loss(y_valid,prediction) # prints 0Even though our model has performed very good (i.e 0 log loss) for the validation dataset, It may not be able to perform well in case on unknown data. In that case, some more steps like hyperparameter tuning, data preprocessing may be needed.
However, the above code may provide information about the basics required to understand logistic regression.
Limitations of logistic regression:
1. Assumes Linear Decision Boundary : Logistic regression assumes a linear relationship between independent variables (features) and the log-odds of the dependent variable (output).It struggles with non-linear relationships unless feature engineering (e.g., polynomial features) is applied.
2. Not Suitable for Complex Relationships : If the data is highly non-linear or has complex interactions, logistic regression may perform poorly compared to models like decision trees or neural networks.
3. Sensitive to Outliers : Logistic regression is sensitive to outliers, especially when using gradient-based optimization.Extreme values can distort the model and affect decision boundaries.
4. Assumes No Multicollinearity : Logistic regression assumes that independent variables are not highly correlated with each other (no multicollinearity). If multicollinearity exists, feature selection or dimensionality reduction (e.g., PCA) is required.
5. Limited to Binary or Multinomial Outputs : Basic logistic regression handles only binary classification (0 or 1). For multiclass problems, one-vs-rest (OvR) or softmax regression (multinomial logistic regression) is required.
To conclude, Logistic regression is one of the powerful and simple machine learning alogirthms. Eventhough it has some limitations, it performs well in most of the cases. Despite its drawbacks, logistic regression is an essential tool in a data scientist’s toolkit, offering a balance between interpretability and predictive power. Understanding its strengths and limitations helps in choosing the right model for a given problem.
