Evaluating a machine learning model
In some articles, it was already explained how to create a model and split the data into train and test. In this session, the metrics for evaluating the model that is created in supervised learning are going to be explained. As known, there are two different algorithms in supervised learning called classification (basically discrete values ) and regression (basically continuous values).
The Evaluation for Classification-Logistic Regression
There are some evaluation metrics for the Classification model. And the most frequently used metrics will be handled.
Confusion Matrix
It is a most used performance measurement machine learning classification problem where the output can be two or more classes. It contains combinations of predicted and actual values.
A confusion matrix or error matrix shows the number of correct and incorrect predictions made by the model compared with the actual classifications in the test set or what type of errors are being made. It is a n*n matrix, where n is the number of classes. This matrix can be generated after making predictions on the test data.
- Accuracy: the proportion of the total number of correct predictions.
- Positive Predictive Value or Precision: the proportion of positive cases that were correctly identified.
- Negative Predictive Value: the proportion of negative cases that were correctly identified.
- Sensitivity or Recall: the proportion of actual positive cases which are correctly identified.
- Specificity: the proportion of actual negative cases which are correctly identified.
- F1-score or F-measure: It is a single metric that combines both Precision and Recall. The higher the F1 score, the better the performance of our model. The range for F1-score is [0,1]. F1 score is the weighted average of precision and recall. The classifier will only get a high F-score if both precision and recall are high. This metric only favors classifiers that have similar precision and recall.
The Evaluation for Linear Regression
Of course, these are quite different metrics than what are used in classification. Here the continuous range instead of a discrete number of in classes is predicted. The reason for that could be explained as follows; we can predict the price of a car as 40K but it could be sold for 43K. It’d be pretty good. But since mostly the binary values are mentioned in classification, it would be tough to say correct-incorrect or good-bad. Therefore the other metrics are used here.
- Mean Absolute Error
MAE is a very simple metric that calculates the absolute difference between actual and predicted values.
Advantages
- The MAE you get is in the same unit as the output variable.
- It is most Robust to outliers.
Disadvantages
The graph of MAE is not differentiable so we have to apply various optimizers like Gradient descent which can be differentiable.
2. R2 coefficient
Get Baris Gül’s stories in your inbox
Join Medium for free to get updates from this writer.
R2 score is a metric that tells the performance of your model, not the loss in an absolute sense that how many wells did your model perform. In contrast, MAE and MSE depend on the context as we have seen whereas the R2 score is independent of context.
3. Mean Square Error
MSE is a most used and very simple metric with a little bit of change in mean absolute error. Mean squared error states finding the squared difference between actual and predicted value.
Advantages
The graph of MSE is differentiable, so you can easily use it as a loss function.
Disadvantages
- The value you get after calculating MSE is a squared unit of output. for example, the output variable is in meters (m) then after calculating MSE the output we get is in meters squared.
- If you have outliers in the dataset then it penalizes the outliers most and the calculated MSE is bigger. So, in short, It is not Robust to outliers which were an advantage in MAE
4. Root Mean Squared Error (RMSE)
It is similar to MSE; just the final value is square-rooted as we calculated the square of errors in MSE.
Advantages of RMSE
- The output value you get is in the same unit as the required output variable which makes interpretation of loss easy.
Disadvantages of RMSE
- It is not that robust to outliers as compared to MAE.
for performing RMSE we have to NumPy NumPy square root function over MSE.
5. Root Mean Squared Log Error
Taking the log of the RMSE metric slows down the scale of error. The metric is very helpful when you are developing a model without calling the inputs. In that case, the output will vary on a large scale.
6. Adjusted R2
The disadvantage of the R2 score is while adding new features in data the R2 score starts increasing or remains constant but it never decreases because It assumes that while adding more data variance of data increases. But the problem is when we add an irrelevant feature in the dataset then at that time R2 sometimes starts increasing which is incorrect.
