Attention is paid to all sorts of advanced computing concepts such as artificial intelligence and deep learning. It is no surprise as with the increase in computing power and where it’s located (e.g., smart phones and the cloud), the ability to perform complex calculations on data in order to derive sophisticated inferences has never had more promise than it does today.

However, one concept often misunderstood is how these sophisticated programs and algorithms get their “smarts.” Sometimes you see claims that AI programs automatically understand your speech or automatically generate meaning from content—all without training. At best, this is misleading, and in some cases, it’s patently false. While it may be true that end users do not have to train some systems or the methods of training are hidden, these systems require training, and often it is significant.

Let’s take, for instance, one of the most famous examples of commercially-available artificial intelligence: IBM Watson. IBM has open APIs that allow organizations and individual developers to tap into Watson’s “cognitive computing” capabilities in order to get some pretty sophisticated capabilities. For example, you can use Watson to determine the presence of trees within a set of images. Or, you can submit personality attributes and get back insights into individual behavior. While Watson presents these capabilities “out of the box,” a significant amount of work has gone into training Watson on how to perform these tasks. End users sometimes participate in these training exercises.

Grounding Out the Truth

Training AI systems starts with development of what is called “ground truth data.” For example, if we have a set of images and we are looking for the presence of a baseball, we present these images along with data on whether a baseball is present. We provide an answer for each image: Image 1 – Baseball Present = False, Image 2 – Baseball Present = True. The system analyzes the images with baseballs and develops inferences into what characteristics represent a baseball. It also analyzes the images without baseballs to develop inferences. The system then processes our set of images and outputs the answers. The results are compared against this ground truth data. When mistakes are made, a user updates the system and the analysis is performed again. If you think about it, this is how humans learn. We learn through example. As children, we learn that “the sky is blue,” not because we develop that insight on our own, but because we are told. And then, we make inferences about what “blue” means.

Social Truthing

An interesting advance in artificial intelligence is the exponential increase in not only computing power, but also access to ground truth data. With “the cloud,” anyone can participate in teaching a system and, again, IBM Watson is a perfect example. Developers not only make use of Watson’s intelligence, they make it “smarter” by submitting new information along with ground truth data. As more contribute, the system gets smarter and everyone benefits. Another way to contribute to general artificial intelligence is during actual use. Imagine a case where end users submit pictures of cars with the intent of getting back the make and model. If the make is correct, but the model is incorrect, the user can make the correction. This validated and corrected data can be submitted along with the image as ground truth data. This constant and expansive feedback loop will have a profound effect with performance improvements.

Machine Learning for Data Extraction

Data location and extraction can also take advantage of social truthing and machine learning. While the current state of the art for “learning” incorporates user feedback on where the data is located on any given document, this information is not typically used for true machine learning. Instead, it is used to create a brittle coordinate-based rule on where the data is located. The next time that document is encountered, the system goes to the location previously provided by the end user. The result is a library of data locations that are very strict and not very flexible. Real machine learning will use this feedback to develop and improve inferences on the most probable locations of needed information and apply it on any document coming in – learning similar to how a person learns. For the future of applied machine learning for data extraction, it’s blue skies ahead.