TLDR: LongRecall is a three-stage framework for accurately evaluating the completeness (recall) of long-form text generated by AI models. It addresses limitations of existing methods by decomposing answers into facts, intelligently selecting candidate matches through lexical and semantic filtering, and then rigorously verifying their alignment using structured LLM-based entailment checks. This approach significantly improves recall accuracy in complex question-answering tasks, offering a more reliable and interpretable evaluation method.
As artificial intelligence (AI) models become increasingly sophisticated in generating long-form text, from reports to complex answers, a critical challenge emerges: ensuring the completeness and factual accuracy of these outputs. In high-stakes fields like medicine and law, missing even a single key fact can have serious consequences. Traditional evaluation methods often fall short, relying on simple word overlap (lexical matching) which can miss valid paraphrases or incorrectly flag irrelevant information. More advanced AI-as-a-judge methods, while capturing broader meaning, can struggle with long texts, leading to misalignments and even AI-generated falsehoods (hallucinations).
To address these limitations, researchers MohamamdJavad Ardestani, Ehsan Kamalloo, and Davood Rafiei have introduced LongRecall, a novel, general three-stage framework designed for robust recall evaluation in long-form text. This structured approach aims to provide a more accurate, interpretable, and scalable way to assess whether AI-generated content adequately covers all required information.
The Three Stages of LongRecall
LongRecall operates through a modular pipeline, breaking down the complex task of recall evaluation into manageable steps:
1. Fact Extraction: The first step involves decomposing both the reference (human-written or authoritative) and the machine-generated answers into self-contained, verifiable facts. This ensures that each piece of information can be understood independently. The framework supports different types of facts, whether they are short anchor phrases with supporting evidence (split-span) or complete statements (monolithic). This stage also includes a coreference resolution step, rewriting pronouns and other anaphoric expressions to make each fact explicit and unambiguous.
2. Candidate Selection: To avoid the computationally expensive task of comparing every reference fact against every generated fact, LongRecall employs a smart candidate selection module. For each reference fact, it identifies a focused subset of generated facts that are most likely to be a match. This is done in two steps: first, lexical filtering uses surface-form similarity (like Fuzzy Jaccard and Longest Common Subsequence) to quickly find close matches. Second, semantic filtering uses dense vector representations and cosine similarity to capture paraphrases and conceptually similar content that might not have direct word overlap. By combining both, LongRecall builds a balanced set of potential matches.
3. Entailment Checking: This is the crucial verification stage. While candidate selection identifies potential matches, it doesn’t guarantee that a generated fact truly ‘entails’ (logically implies or covers) the reference fact. LongRecall uses large language models (LLMs) with two prompting strategies for this: one-to-one verification for single strong candidates, and multiple-choice entailment when several candidates might jointly cover a reference fact. This structured LLM-based verification ensures that only genuinely covered facts are counted, significantly reducing false positives and false negatives.
A Unified Workflow
The LongRecall pipeline integrates these stages into a “verify-as-you-go” workflow. It first uses the efficient lexical filtering and immediate entailment checks. If a reference fact is covered, processing stops for that fact. If not, it moves to the more resource-intensive semantic filtering and a second round of entailment checking. This staged approach minimizes computational overhead while maintaining high accuracy.
Also Read:
- FINCH-ZK: Enhancing Factual Reliability in Large Language Models
- DEPTH: A New AI Framework for Accurate Relation Extraction and Halucination Reduction
Demonstrated Superiority
The researchers evaluated LongRecall on three challenging long-form question-answering benchmarks: QAMPARI, RoMQA, and ExpertQA. The results show that LongRecall consistently and substantially outperforms existing lexical baselines and holistic “LLM-as-a-Judge” methods. For instance, on the ExpertQA dataset, where lexical methods almost entirely failed, LongRecall achieved significantly higher F1 scores. Even when using the same underlying LLM, LongRecall’s structured approach enabled it to measure recall much more effectively, drastically reducing errors like self-mapping and spurious topical matches that plague holistic methods.
While the framework is robust, error analysis revealed that most performance variations stem from the entailment checking stage, highlighting the trade-offs between different LLMs (e.g., Gemini being more permissive, Qwen being stricter). Future work will focus on refining candidate selection for domain-specific terminology and fine-tuning entailment models.
LongRecall represents a significant step forward in evaluating the completeness of AI-generated text, offering a foundational framework for developing more reliable and trustworthy AI systems across various applications. You can read the full research paper here.
