Error Analysis

Error Analysis is the process of manually identifying error in the training data or the learning algorithm so as to improve the accuracy of the model.

If we suspect error in the data, we could take a random sample of 100 images (for an image classification task) from the dev set and find out the impact of the error on the final accuracy. Accordingly, we could decide if we wanted to invest time and effort in reducing the error.

For example, say that the dev set error is 10%. We find out that 0.6% is because of incorrectly labeled data. In such a case, it is more important to find out the cause of the remaining 9.4% error instead of fixing the labels of the incorrectly labeled data.