Secret Life of a Confidence Score | Knowledge Base | Definition
What Is A Confidence Score? How Is It Assigned?
A confidence score essentially is a number assigned to a task, but it’s not just a number. We derive the number from the intelligent document processing system. The confidence score typically ranges between 0-100, they might go beyond a hundred. There are some systems that have maximum confidence scores of 1,500. Ultimately, competence scores come from the software that outputs them and are associated with a specific task. Now these specific tasks can be OCR so the task of converting characters and words within an image to text that is machine-readable. That’s one example. It could be something more specialized like training machine learning algorithms to understand the difference between one document within a loan file from another document. It’s very specific types of tasks where you are training the software on a large amount of data. The sample sizes are significant. Most often the data is hundreds of thousands.
Data scientists then analyze the output of these machine learning algorithms and they’ll make adjustments to the input data to emphasize certain types of use cases over others. They also might adjust the algorithms. They make adjustments to how things are weighted with the goal of improving the performance in terms of accuracy. Let’s say the system output/results are 80% correct for a document classification task. Their goal is ultimately to optimize to the greatest extent possible the overall accuracy of that output data. If they’re doing it right, they’re also optimizing the reliability of the confidence scores associated with the tasks — those numbers that go with each type of automated task.
So ultimately the objective is to create a high level of correlation. What does that mean? It means that we want confidence scores to be higher when the data is accurate and we want confidence scores to be lower when the data is not accurate. They focus on both of these: data accuracy and the corresponding core confidence score. If a lot of the data output is accurate with low confidence scores, that’s a problem. This requires adjustments to ensure the confidence scores match the level of data accuracy so that when in production the intelligent automation system provides straight through processing for the accurate data results and sends only the low confidence score results for further verification.