TLDR: A study on AI-assisted data extraction found that AI’s ‘inaccuracies’ are mostly due to the interpretability of questions, not hallucinations. AI was highly consistent for concrete questions, and human-human consistency showed similar variability. Humans were more prone to factual errors than AI. Repeating AI extractions can help identify interpretive questions, making AI a valuable partner in knowledge synthesis.
In the realm of health professions education, synthesizing knowledge from numerous publications, often referred to as literature reviews, is a cornerstone for advancing theory and practice. However, this process, particularly the data extraction phase, is notoriously labor-intensive. The advent of Artificial Intelligence (AI) promises to streamline this, yet it introduces a critical question: how do we differentiate between AI-generated “hallucinations”—incorrect or fabricated content—and legitimate interpretive differences arising from subjective judgment?
Understanding AI’s Role in Data Extraction
A recent study delved into this very challenge, developing an AI-powered platform called MAKMAO (Machine-Assisted Knowledge extraction, Multiple-Agent Oversight) to automate data extraction. The researchers evaluated the accuracy and error rates of AI by comparing its extractions to those done by humans across 187 publications and 17 specific extraction questions from a previously published review. They meticulously measured consistency between AI and humans, between different humans, and even between repeated AI extractions, using established methods like interrater reliability for clear-cut answers and thematic similarity for open-ended responses.
Key Findings: Interpretation Over Hallucination
The study’s findings offer significant insights. AI extraction showed high consistency with human responses for straightforward questions where answers were explicitly stated in the papers, such as titles, study aims, and results. However, consistency was lower for questions that demanded subjective interpretation or were not explicitly addressed, like classifying outcomes using Kirkpatrick’s levels or understanding the study’s rationale for using AI.
Remarkably, the consistency between different human extractors was not higher than that between AI and humans. In fact, human-human consistency mirrored the same pattern of variability depending on the question type. This suggests that the difficulty in consistently answering certain questions is inherent to the questions themselves, rather than a limitation of AI.
When analyzing the discrepancies between AI and human responses (about 24.2% of all cases), the researchers found that actual AI inaccuracies accounted for a very small fraction (only 1.51%). These inaccuracies were typically omissions or major misclassifications, but notably, AI did not generate “factual lapses”—assertions that were not supported by the source text. In contrast, human extractors were nearly three times more likely to make inaccuracy errors (4.37%), including factual lapses and misclassifications.
The vast majority of AI-human discord (18.3%) was attributed to interpretive differences. This included instances where multiple correct answers were possible, varying levels of detail were provided, or minor disagreements in classification occurred. This highlights that AI’s “accuracy” is largely influenced by the interpretability of the question, rather than a tendency to fabricate information.
Also Read:
- When Less is More: Enhancing Trust in AI by Obscuring Less Factual Content
- Unpacking User Engagement with AI Explanations: A Surprising Look at Trust and Decision-Making
Implications for Research and Trust in AI
The study suggests that AI can be a transparent and trustworthy partner in knowledge synthesis. The ability of AI to identify questions prone to interpretation, even without human comparison (by observing AI-AI consistency), is a powerful tool. If repeated AI extractions for the same question yield varied answers, it signals either high interpretability or potential ambiguity in the question itself. This allows researchers to refine their questions, provide clearer definitions, or acknowledge where human interpretive expertise is crucial.
This approach moves AI beyond being just a time-saver. By leveraging AI to identify and clarify interpretive complexities, researchers can sharpen their extraction questions and focus human effort where it truly adds value—in nuanced interpretation and critical contextual insights. The full research paper can be found here.
