TLDR: A research paper from the University of North Texas presents ClaimIQ, a system for verifying numerical and temporal claims using retrieved evidence. It compares zero-shot prompting with instruction-tuned LLMs and supervised fine-tuning with LoRA, alongside various evidence selection strategies. While fine-tuned LLaMA with LoRA showed strong performance on validation data, a significant drop on the test set highlighted generalization challenges, especially for “True” and “Conflicting” claims, emphasizing the need for robust evidence processing and model adaptation.
In an era where digital misinformation spreads rapidly, the ability to automatically verify factual claims, especially those involving numbers or dates, has become critically important. Researchers from the University of North Texas presented their system, ClaimIQ, at the CLEF 2025 CheckThat! Lab, focusing on tackling this challenge by verifying numerical and temporal claims using retrieved evidence.
The core of their work, detailed in their paper “ClaimIQ at CheckThat! 2025: Comparing Prompted and Fine-Tuned Language Models for Verifying Numerical Claims”, explores two main strategies: using large language models (LLMs) with direct instructions (zero-shot prompting) and fine-tuning LLMs with specific training data using a technique called LoRA (Low-Rank Adaptation). They also investigated how the quality and selection of evidence impact the verification process, experimenting with different ways to feed information to the models.
Approaches to Claim Verification
The team explored two distinct paths. The first involved “zero-shot prompting,” where instruction-tuned LLMs like LLaMA were given a claim and evidence, then asked to classify it as True, False, or Conflicting without any prior task-specific training. This method tests the LLM’s inherent understanding and generalization capabilities.
The second approach was “supervised fine-tuning,” which involved training models like RoBERTa and LLaMA (with LoRA) on a dataset of claims and their correct classifications. LoRA is a parameter-efficient method that allows for effective fine-tuning of large models without requiring extensive computational resources.
Enhancing Evidence Quality
A crucial aspect of verifying claims is the quality of the evidence provided. The researchers tested several strategies to improve the “signal-to-noise ratio” of the input evidence. These included using the Full Document, which is the entire top-ranked document retrieved by the BM25 algorithm. While comprehensive, this can include irrelevant information. Another strategy was selecting the Top-3 BM25 Sentences, which involved picking the three most relevant sentences from the top documents based on lexical matching with the claim. For improved semantic understanding, they also used Top-3 MiniLM Sentences, employing a semantic embedding model (MiniLM) to find the three sentences most semantically similar to the claim, capturing deeper meaning.
Model Performance and Key Findings
The experiments were conducted on the English subset of the CLEF 2025 Task 3 dataset, which includes over 15,000 claims. On the validation set, the fine-tuned LLaMA model with LoRA, using full-document evidence, achieved the highest performance, demonstrating a strong ability to classify claims accurately across True, False, and Conflicting categories.
However, a significant observation emerged when evaluating the models on the test set. While the fine-tuned LLaMA with LoRA still performed best among all configurations, there was a notable drop in overall performance compared to the validation results. This “generalization challenge” suggests that models might struggle to adapt to new, unseen evidence structures or language styles in real-world scenarios. Specifically, classifying “True” and “Conflicting” claims remained particularly difficult on the test set, while “False” claims were more consistently identified.
Also Read:
- Improving RAG Performance with Hierarchical Document Chunking
- Teaching LLMs to Trust Context: The SI-FACT Framework
Looking Ahead
The findings underscore the importance of both the model’s architecture and the quality of the evidence in building robust fact-verification systems. While large language models, especially when fine-tuned efficiently, show great promise, challenges remain in ensuring they generalize well to diverse and potentially noisy real-world data. Future research will focus on improving model robustness through domain-adaptive training, more sophisticated evidence retrieval, and reasoning-aware modeling strategies to bridge the gap between validation and real-world test performance.
