Confusion Matrix

What is confusion matrix?

A confusion matrix is a measure used to solve classification problems. It can be put to use against binary classification problems as well as multiclass classification problems. You could say that a confusion matrix is essentially a technique that can be used for summarizing the performance of a classification algorithm.

If you only look at classification accuracy, you might not get the right picture, as there is the possibility of it being misleading in situations where you have an unequal number of observations in each class or you have upwards of two classes in your dataset.

The confusion matrix is essentially a table that describes the performance of a classification model by using a test dataset for which the actual values are known. The table contains four different combinations of predicted and actual values.

A confusion matrix is very useful to measure Precision, Recall, Specificity, and Accuracy. It is also used to measure AUC-ROC curves.

If you calculate a confusion matrix, you would get a better idea of what your classification model is getting right and what errors it is making. To put this simply, the confusion matrix shows you the manners in which your classification model is getting confused when it makes predictions. But it is not limited to simply showing you what errors your classification model is making. It goes deeper to show you what types of errors your classifier is making - this is information that is more important and actionable. This breakdown helps you overcome the limitations involved in making use of classification accuracy on its own.

A confusion matrix is basically a summary of the prediction results on a classification problem. The number of correct and incorrect predictions gets summarized with count values and is broken down by each class. This is the key to the confusion matrix.

What is classification accuracy and what are its limitations?

The classification accuracy is the ratio of correct predictions to total predictions made by the classifier. It is usually expressed as a percentage. 

Here is the formula that you can make use of to calculate the classification accuracy:

Classification accuracy = (Correct predictions / total predictions) x 100

If you invert the value, you can very easily turn the classification accuracy ​​into a misclassification rate or error rate. The formula for the error rate would be:
Error rate = [1 - (Correct predictions / Total predictions)] x 100

While classification accuracy could be good to start off with, you would usually face an number of problems in practice with it. The main limitation of classification accuracy is that it does not show you the details that you need to better understand the performance of your classification model. There are two scenarios in which you would be likely to encounter this problem:

• The first is when your data has more than 2 classes. With 3 or more classes you could get a classification accuracy of 80%, but you don’t know if that is because all classes are being predicted equally well or whether one or two classes are getting neglected by the model.
• The second is when your data doesn’t have an even number of classes. You could achieve accuracy of 90% or more, but this is not a good score if 90 records for every 100 belong to one class and you could achieve this score by always predicting the most common class value.

So the biggest problem with classification accuracy is that it doesn’t give you the detail you need to diagnose the performance of your model, but you could get this detail through the use of a confusion matrix.

How do you calculate a confusion matrix?

You will first require a validation dataset (test dataset), the actual outcomes of which you already know. 

Next, you will need to make predictions for every row in your validation dataset.

After doing that, compare the predicted outcomes and the actual outcomes to count the number of accurate predictions for every class and the number of inaccurate predictions for every class.

The numbers are then organized into a table (matrix) as shown above, with every row of the table corresponding to a predicted class and every column of the matrix corresponding to an actual class.

Now the total number of correct and incorrect classifications are filled into that table.

What are the outputs of a confusion matrix?

True Positive: This is when the predicted result and the actual result are both positive.

False Positive: This is when the predicted result is positive, but the actual result is negative.

True Negative: This is when the predicted result and the actual result are both negative.

False Negative: This is when the predicted result is negative but the actual result is positive.

Confusion matrices have two types of errors. 

False Positives are considered Type I errors, while False Negatives are called Type II errors.

‍

Which terms are used in confusion metrics?

There are five metrics that help us understand the validity of our model. 

Accuracy: How often does the classifier generate the correct output?

Formula: (True Positives + True Negatives)/Total

Misclassification: How often does the classifier generate the wrong output?

Formula: (False Positives + False Negatives)/Total

This is also known as the Error Rate

Recall (or Sensitivity): How often does it predict a positive outcome when the actual outcome is positive?

Formula: True Positives/(True Positives + False Negatives)

Specificity: How often does it predict a negative outcome when the actual outcome is negative?

Formula: True Negatives/(True Negatives + False Positives)

Precisions: When the classifier predicts a positive outcome, how often is it correct?

Formula: True Positives/(True Positives + False Positives)

What is the ROC curve?

An ROC curve is a graph summarizing the performance of a classifier over all possible thresholds.

You can generate the ROC curve by plotting the True Positive Rate against the False Positive Rate.

True Positive Rate = Recall or Sensitivity

False Positive Rate refers to how often the model predicts a positive outcome when the actual outcome is negative.

Formula: False Positive/(True Negative + False Positive)