TLDR: This research introduces a hybrid fact-checking system that integrates Knowledge Graphs (KGs), Large Language Models (LLMs), and web search agents. The system first uses KGs for rapid, precise lookups, then an LLM for classification. If information is insufficient, a web search agent retrieves evidence, which is then re-evaluated. This modular, zero-shot approach achieved an F1 score of 0.93 on the FEVER benchmark and successfully found evidence for over 70% of claims previously labeled as ‘Not Enough Information’, enhancing both coverage and interpretability in claim verification.
In today’s information-rich world, distinguishing fact from fiction is more critical than ever. Automated fact-checking systems are essential, but they face significant challenges. Large Language Models (LLMs), while excellent at generating human-like text, can sometimes produce unverified or even false information, a phenomenon often called hallucination. On the other hand, traditional knowledge-graph-based fact-checkers offer precise and clear evidence but are limited by the scope of their data and can be slow in real-time scenarios.
A new research paper introduces a hybrid fact-checking approach that aims to overcome these limitations by combining the strengths of Knowledge Graphs (KGs), Large Language Models, and real-time search agents. This innovative system is designed to make claim verification more reliable and interpretable.
How the Hybrid System Works
The system operates in three distinct, autonomous steps:
1. Knowledge Graph (KG) Retrieval: The first step involves a rapid lookup in DBpedia, a large knowledge base extracted from Wikipedia. This stage quickly checks for direct, one-hop connections within the structured data to verify claims. It’s designed for speed and precision, leveraging the explicit and transparent evidence KGs provide.
2. LM-based Classification: If the Knowledge Graph provides sufficient information, an LLM (specifically, GPT-4o mini or DeBERTa-v3 MNLI) classifies the claim as ‘Supported’ or ‘Refuted’ based on the KG evidence. This step uses a task-specific prompt to guide the LLM’s decision-making process.
3. Web Search Agent: This is a crucial fallback mechanism. If the KG retrieval and initial LM classification return ‘Not Enough Information’ (NEI), a Web Search Agent is activated. It paraphrases the original claim into several search queries and uses a search API (like Google Programmable Search) to retrieve relevant web snippets. These snippets are then re-evaluated by the same LLM or DeBERTa model to determine if the claim is ‘Supported’ or ‘Refuted’. This ensures broader coverage for claims not easily verifiable through structured knowledge bases.
The modular design of this pipeline means that different components can be easily tested or replaced, making it flexible and adaptable. The system does not require specific fine-tuning for each task, operating effectively in a zero-shot inference mode.
Impressive Results and Addressing ‘Not Enough Information’ Claims
The researchers evaluated their approach on several benchmarks, including FEVER, FEVER 2.0, and FactKG. The pipeline achieved an F1 score of 0.93 on the FEVER benchmark for the Supported/Refuted split, demonstrating strong performance without task-specific fine-tuning. This F1 score is competitive with, and in some cases surpasses, other state-of-the-art systems.
One of the paper’s significant contributions is its targeted study on claims initially labeled as ‘Not Enough Information’ (NEI). The hybrid system frequently uncovered valid evidence for claims that were originally deemed unverifiable. A reannotation study, involving both expert human annotators and LLM reviewers, confirmed that in over 70% of these NEI cases, the system successfully retrieved meaningful evidence. This highlights the pipeline’s ability to expand coverage and provide verification for claims that might otherwise be dismissed.
Also Read:
- Enhancing AI Agent Safety and Trustworthiness with a New Response Framework
- MemSearcher: Enhancing LLM Search Agents with Smart Memory Management
Why This Matters
This hybrid fact-checking system offers a robust solution to the challenges of automated claim verification. By prioritizing the precision and interpretability of Knowledge Graphs and falling back to the broad coverage of web retrieval when necessary, it delivers a balanced approach. The system’s ability to find evidence for previously unverified claims, coupled with its modular and open-source nature, makes it a valuable tool for improving the reliability and transparency of information verification. For more details, you can read the full research paper here.
