TLDR: DecMetrics introduces three new metrics (Completeness, Correctness, Semantic Entropy) to automatically assess the quality of decomposed claims from large language models (LLMs). It also presents DecModel, a lightweight decomposition model optimized using these metrics, and Claim2Atom, a new benchmark, all aimed at enhancing the reliability of fact-checking systems for LLM-generated text by ensuring high-quality claim decomposition.
Large language models (LLMs) are increasingly used to generate long-form text, but these outputs often contain factual inaccuracies. To address this, fact-checking systems are crucial, and a foundational component of these systems is claim decomposition. This process involves breaking down complex claims into simpler, individual “atomic claims” that can be more easily verified.
However, existing research has primarily focused on generating these decomposed claims, with less emphasis on evaluating their quality. This gap is significant because the quality of decomposed claims directly impacts the accuracy of the overall fact-checking process. If claims are poorly decomposed—missing information, containing errors, or being redundant—the final fact-checking score can be misleading.
Introducing DecMetrics: A New Standard for Claim Evaluation
To bridge this critical gap, a new framework called DecMetrics has been introduced. DecMetrics comprises three novel metrics designed to automatically assess the quality of atomic claims produced by decomposition models:
- COMPLETENESS: This metric evaluates whether the collection of decomposed atomic claims collectively covers all necessary aspects of the original, complex claim. It ensures that no essential information is lost during the decomposition process.
- CORRECTNESS: This metric assesses whether each individual atomic claim is factually faithful to the original claim, ensuring that no fabrication or hallucination has occurred. A high correctness score indicates minimal factual deviation.
- SEMANTIC ENTROPY: This metric quantifies the diversity and independence of atomic claims. It aims to identify and penalize repetitive paraphrasing or semantic overlap, encouraging distinct and non-redundant decompositions.
These three metrics function similarly to recall, precision, and accuracy in traditional evaluation, providing a comprehensive assessment of a decomposition model’s strengths and weaknesses. For instance, a model that simply returns the original claim would score high on completeness and correctness but zero on semantic entropy, highlighting its failure to decompose effectively.
DecModel: Optimizing Decomposition with Rewards
Building on these new evaluation metrics, the researchers also developed DecModel, a lightweight claim decomposition model. This model is optimized using a reinforcement learning framework, where the DecMetrics (COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY) serve as reward signals. By integrating these metrics into the training process, DecModel is designed to generate atomic claims that inherently align with high-quality standards.
Claim2Atom: A Comprehensive Benchmark
To further facilitate rigorous evaluation and future research in claim decomposition, a new benchmark called Claim2Atom has been proposed. This benchmark aggregates and filters existing public datasets, FActScore and WICE, and combines them with a newly curated dataset called DecData. Claim2Atom provides a robust framework for assessing the capabilities of various LLMs and specialized decomposition models.
Synthetic Data Generation for Training
To train the models that power DecMetrics, a structured synthetic data generation process was employed. This involved sampling entities from Wikipedia, extracting their summaries as original claims, and then using LLMs (specifically Qwen 3-32B) to iteratively decompose these claims into non-splittable atomic claims. A reverse-checking process, also guided by an LLM, verified the completeness, correctness, and independence of these decomposed claims, creating both supported and unsupported examples for training.
Also Read:
- Unveiling LLM Uncertainty: A New Method for Detecting Hallucinations in Black-Box Models
- DeepTRACE: A Framework for Auditing AI Search and Research Systems
Performance and Impact
The fine-tuned DeBERTa-v3-large model, trained on the synthetic DecMetrics dataset, demonstrated superior performance on its specific task compared to general Natural Language Inference (NLI) models. DecModel itself showed competitive performance with much larger LLMs while being significantly more parameter-efficient. This indicates that DecMetrics and DecModel offer a promising approach to enhancing the reliability and effectiveness of fact-checking systems for AI-generated content.
While the structured approach to synthetic data generation is effective, the researchers acknowledge limitations, including potential generalization issues to real-world complexities and the resource-intensive nature of the decomposition and verification process. Nevertheless, DecMetrics, DecModel, and Claim2Atom represent significant advancements in ensuring factually consistent outputs from large language models. You can read the full research paper here: DecMetrics: Structured Claim Decomposition Scoring for Factually Consistent LLM Outputs.
