TLDR: HAMLET is a new, automated framework for evaluating how well Large Language Models (LLMs) understand very long texts, like books. It breaks down text into a hierarchical “key-fact tree” (root, branch, leaf) and uses query-focused summarization to test LLMs’ recall and factual accuracy at different detail levels. Validated against human experts with over 90% agreement and 25x cost reduction, HAMLET reveals LLMs struggle with fine-grained details, are affected by the “lost-in-the-middle” problem, and often hallucinate. It also shows performance gaps between open-source and proprietary models, and larger models generally perform better.
Large Language Models (LLMs) are becoming increasingly capable of processing and understanding very long texts, even entire books. However, evaluating how well these models truly comprehend such extensive content has been a significant challenge. Traditional evaluation methods often fall short, either focusing on short-form tasks or only assessing a shallow understanding of the text. This is where a new framework called HAMLET comes in, offering a comprehensive and automated way to assess LLMs’ multi-level comprehension in book-length contexts.
Introducing HAMLET: A New Approach to LLM Evaluation
Developed by researchers at the Korea Advanced Institute of Science and Technology, HAMLET (Holistic and Automated Multi-Level Evaluation for Long Text) addresses the limitations of previous evaluation benchmarks. It’s designed to go beyond surface-level understanding, probing how well LLMs can recall and accurately represent information at different levels of detail within a long document.
The core of HAMLET is its innovative “key-fact tree” structure. Imagine a book’s content broken down into a hierarchy: a “root” representing the main theme of a section, “branches” for supporting ideas, and “leaves” for fine-grained details. HAMLET automatically constructs these key-fact trees from manageable 4K-token segments of a book. This hierarchical structure allows for the creation of “detail-aware queries” that can assess an LLM’s ability to extract information from high-level concepts down to specific facts.
How HAMLET Works
The framework operates in three main stages:
1. Query Construction: Using the key-fact trees, HAMLET generates specific queries. These queries can be either “analytical,” focusing on deeper meaning and thematic interpretation, or “narrative,” emphasizing story progression and key events. This dual approach ensures a broad assessment of comprehension.
2. Summary Generation: LLMs are given the entire book (up to 114,000 tokens) and asked to generate a summary in response to each query. This task evaluates the LLM’s ability to extract all relevant information (recall) and produce factually accurate responses (faithfulness) over long documents.
3. Summary Evaluation: This is where HAMLET truly shines with its automation. Instead of relying on costly and often inconsistent human evaluation for the entire book, HAMLET uses LLM-based evaluators (specifically GPT-4o) to assess the generated summaries. It checks for “multi-level recall” (how well key-facts are captured at root, branch, and leaf levels) and “multi-level faithfulness” (the accuracy of the content without hallucinations at each level). This automated pipeline has been rigorously validated, achieving over 90% agreement with expert human judgments while significantly reducing costs by up to 25 times.
Key Findings from HAMLET’s Benchmarking
Using HAMLET, researchers benchmarked eight high-performing LLMs, revealing several important insights:
- Struggle with Fine-Grained Details: LLMs consistently showed a decline in recall from high-level themes (root) to fine-grained details (leaf), indicating difficulty with precise, detailed comprehension. This was more pronounced for analytical content than narrative content.
- Lost-in-the-Middle Effect: The framework confirmed the “lost-in-the-middle” effect, where LLMs struggle to recall information from the middle sections of long inputs. This effect was particularly sharp for leaf-level (detailed) information.
- Proprietary vs. Open-Source: Proprietary models generally exhibited higher recall across all abstraction levels compared to open-source models, though open-source LLMs were competitive at the root-level.
- Model Scale Matters: Larger LLMs consistently outperformed smaller ones in long-context comprehension tasks, although increasing model size alone didn’t mitigate the lost-in-the-middle effect.
- Hallucinations are Common: Faithfulness scores were generally low across all models, indicating a frequent tendency for hallucinations, especially for content not directly aligned with the key-fact tree.
- Reasoning Models Trade-off: Surprisingly, reasoning-optimized models showed a decline in recall but an improvement in faithfulness, suggesting a trade-off where prioritizing inference might hinder information extraction but lead to more factually accurate responses.
Also Read:
- Assessing Language Models’ Grasp of Game States Through Chess
- Unpacking AI’s Inner Monologue: When Does Chain-of-Thought Truly Guide, and When Does It Justify?
The Future of Long-Context LLM Evaluation
HAMLET represents a significant step forward in evaluating LLMs’ ability to understand book-length texts. Its automated, multi-level approach provides a scalable and reliable benchmark that can be extended to new domains and languages in the future. The framework’s code and dataset are publicly available, fostering further research and development in this critical area of AI. For more details, you can read the full research paper here.
