Problem-solving in the Age of IDP
Have you ever experienced starting-off solving a particular problem, only to find that once it is tackled, that there is yet another perquisite problem to solve? And once again when solved, still another and another? For instance, I need to put in a fence to keep my puppy on my property (problem 1), but then find that the land has a number of boulders, which first must be removed (problem 2). Before I address that, I find that I need to have utility lines marked and potentially moved (problem 3). Most likely all of us have experienced some form of this endless cycle of problem-solving so you can appreciate the challenges associated with the adoption of intelligent document processing (IDP), also known as intelligent capture.
Your organization might find that it is losing business from prospective customers abandoning a loan application process due to lack of ability to quickly interact with your prospects. When you go and try to increase interactivity, you find that your team cannot keep up with the number of loan applicants. When you try to solve the problem of workflows that cause the delays, you find that to remove delays requires you to automate the process of loan documentation intake, which includes document review and data entry. And then when you go to solve that problem, you find that the solution you selected requires a significant amount of training and professional services expense. After addressing that problem with an increase in budget, you find that a large amount of sample data is required to properly configure and tune a system.
Addressing the Complexities
There is always a problem to be solved. So how do we get around all of this complexity, especially if the most complex problems – namely, that task of comprehensively and accurate converting document-based information into structured data – are only prerequisites to solve the real problems associated with improved customer experiences, more-efficient and controllable processes, etc.?
Put another way, should the prerequisite problems of document automation be the biggest hurdles with an initiative to improve customer experience? The answer is an definitive, “NO!” And yet, that is where we find ourselves in 2020. Complex software that solves only one part of the overall problem.
Answers in the Data
The answer lies in turning our attention not to the challenges of how to learn, configure, optimize and manage complex IDP software, but to the processes involved with obtaining good quality input data. With that we let the system manage all the complexity. The objective is to remove the man months or years performing manual analysis, configuration and optimization.
After all, isn’t that what machine learning is all about? Isn’t it about letting the machines do all the heavy lifting while we focus on defining the problems and using the output? In the case of IDP, defining the problem involves identification of target data and the associated sample sets that describe it. If we do that correctly, machine learning algorithms can identify the best way to handle common tasks such as document identification (and separation), or locating target data and automatically validating it.
Problem-solving with Software-defined Systems
Originally emerging within communications and network infrastructure, software-defined systems allow traditional hardwired networks to be defined and optimized by software, making them much more flexible and easier to manage. This concept is catching on in other sectors. For IDP, a software-defined system can configure itself using only tagged sample data. There is no need to perform programming, write scripts, use regular expressions or even define tag-value pairs.
The software accomplishes it all from a simple set of documents and the data values that are desired. Input sets can be a simple list of files and their actual document classes. Or, it can be a list of files and the values “trapped” within unstructured text that are needed to perform a transaction. From there, the software analyzes all the attributes. This includes the clearly visible ones as well as those that are mostly hidden (e.g., have you ever noticed and used pixel-level distance as a parameter to find a data field?) in order to define the most reliable machine learning model to perform a task.
Software-defined systems can also analyze the results and identify how to understand when data is most reliable and when assistance is needed. All of this can be achieved as a “compute-time” activity measured in hours, not days or weeks.
All of this can be achieved as a “compute-time” activity measured in hours, not days or weeks.
The promise of software-defined systems of all types is to make much more adaptable and reliable systems without the traditional costs. For IDP, the result is more data at a higher rate of accuracy, all with significantly less effort. Since IDP is one prerequisite for solving many larger problems, the advent of software-defined IDP promises to usher-in a new level of automation.