TLDR: LiteraryQA is a new, high-quality dataset derived from NarrativeQA, specifically focusing on literary works. It addresses issues like noisy documents and flawed question-answer pairs in existing benchmarks through a human and LLM-validated refinement process. A meta-evaluation shows that traditional metrics poorly correlate with human judgment, while LLM-as-a-Judge evaluations, especially when provided with summaries, align strongly with human preferences, making LiteraryQA a more reliable benchmark for evaluating long-document narrative AI.
Evaluating how well Artificial Intelligence (AI) systems understand long, complex stories has always been a significant challenge. Current methods often struggle with the nuances of narrative texts, which unfold across many pages and involve intricate sequences of events and character interactions. A widely used benchmark for this, called NarrativeQA, has been found to have its own set of problems, including messy documents and flawed question-answer pairs, making it difficult to truly assess an AI’s comprehension abilities.
To address these issues, researchers have introduced a new, high-quality dataset called LiteraryQA. This dataset is a carefully refined subset of NarrativeQA, focusing exclusively on literary works. The goal of LiteraryQA is to provide a more reliable and accurate way to evaluate how well AI models can perform Question Answering (QA) on long narrative documents.
How LiteraryQA Was Created
The creation of LiteraryQA involved a meticulous, multi-step process that combined human expertise with the power of Large Language Models (LLMs). The pipeline focused on two main areas: document quality and question-answer pair quality.
First, at the document level, the team identified and removed documents that were mismatched with their summaries, theatrical plays, and other non-narrative texts. They also implemented a sophisticated text cleaning algorithm to strip away extraneous content like HTML artifacts, Project Gutenberg headers, and legal sections, which often inflate document length and can confuse AI models. This cleaning process, on average, made documents about 3,000 tokens shorter, representing a significant reduction in noise.
Second, at the QA-level, the process focused on individual question-answer pairs. This involved identifying and removing duplicate questions, and then refining questions and answers that were either grammatically incorrect, ill-posed (meaning they contained false assumptions or were unanswerable from the summary), or invalid (factually inaccurate or incomplete). An LLM, specifically Claude 3.5 Haiku, played a crucial role in identifying and correcting these issues, with human annotators validating its outputs. This rigorous refinement led to modifications in 38% of the QA samples, significantly enhancing the dataset’s quality.
Evaluating the Evaluation Methods
A key part of this research was a meta-evaluation of different metrics used to score AI answers. The findings were quite revealing. Traditional n-gram-based metrics, such as ROUGE-L, F1, and Exact Match (EM), showed a generally poor correlation with human judgment, especially on the original NarrativeQA dataset. This means these metrics often failed to agree with how humans would rate an AI’s answer, giving low scores to semantically correct responses that simply used different wording or had minor typos.
However, one n-gram metric, METEOR, performed moderately better due to its ability to handle stemming and synonyms. More importantly, evaluations using an “LLM-as-a-Judge” paradigm (where another LLM scores the AI’s answer) demonstrated a significantly higher correlation with human judgments. This was particularly true when the judge LLM was also given the book’s summary as additional context, allowing it to assess answers more broadly than just against specific reference answers. The study found that LLM-as-a-Judge, especially in this “summary-based” setting, is the most reliable method for evaluating narrative QA.
Benchmarking AI Models
The researchers also benchmarked several long-context LLMs on LiteraryQA, including both open-source and closed-source models. They tested these models in different settings: “open-book” (with access to the full text), “closed-book” (relying only on pre-training knowledge), and “summary” (with access to story summaries). While LiteraryQA’s higher quality led to better scores across all metrics and a stronger reflection of human judgments, the overall performance of even state-of-the-art models remains below that seen in other QA settings. This indicates that open-ended narrative QA, even with a refined dataset like LiteraryQA, continues to be a very challenging task for current AI systems.
Also Read:
- NARRABENCH: A New Framework to Assess AI’s Grasp of Stories
- Unpacking LLM Judge Capabilities: Human-Like vs. Super-Consistent AI
Looking Ahead
LiteraryQA represents a significant step forward in creating a more reliable and fair benchmark for evaluating AI’s ability to understand complex narratives. By addressing the noise and flaws in previous datasets, it enables a clearer assessment of model capabilities. While challenges remain, particularly in the cost and transparency of LLM-as-a-Judge evaluations, this work paves the way for future research into deeper narrative comprehension in AI. For more details, you can read the full research paper: LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA.
