TLDR: The OOLONG benchmark evaluates large language models’ ability to reason and aggregate information over long contexts, moving beyond simple retrieval tasks. It comprises OOLONG-synth (synthetic classification, counting, user, and temporal tasks) and OOLONG-real (tasks over Dungeons & Dragons transcripts). Findings show that even advanced models like GPT-5 and Claude-Sonnet-4 achieve less than 50% accuracy at 128K context, indicating significant challenges in effectively utilizing long context for complex aggregation, especially with temporal reasoning.
As large language models (LLMs) continue to expand their context windows, a critical question arises: are these models truly utilizing the vast amounts of information provided, or are they merely performing simple retrieval tasks? A new research paper introduces OOLONG, a benchmark designed to rigorously evaluate LLMs’ capabilities in long-context reasoning and information aggregation.
The paper, titled “OOLONG: Evaluating Long Context Reasoning and Aggregation Capabilities,” highlights that many existing long-context evaluations often rely on tasks where most of the context can be disregarded as noise, focusing primarily on retrieval. OOLONG aims to fill this gap by requiring models to analyze individual text chunks and then aggregate these analyses to answer complex distributional questions.
Introducing OOLONG: A Two-Part Benchmark
OOLONG is divided into two distinct task sets:
-
OOLONG-synth: This set features naturalistic synthetic tasks built from existing in-context learning datasets. It allows researchers to precisely control and isolate different components of the reasoning problem. Tasks here involve implicitly labeling examples within the context and then reasoning over distributional properties of these labels. This includes counting tasks (e.g., identifying the most frequent label), user-specific patterns (e.g., which user is represented most often), and temporal reasoning (e.g., changes in label distribution over time).
-
OOLONG-real: This part of the benchmark uses real-world conversational data, specifically transcripts from live-action Dungeons & Dragons shows. Unlike the synthetic tasks, OOLONG-real presents challenges that cannot be easily broken down into simple, independent parts. It requires models to reason about character states, campaign statistics, dice rolls, and spells cast over long, unscripted conversations, using human-annotated gold answers for evaluation.
Both OOLONG-synth and OOLONG-real demand multi-step reasoning. Models must identify relevant segments, classify or categorize them, and then aggregate these individual decisions to produce a final answer. The tasks are designed to be individually simple, ensuring that the benchmark measures long-context reasoning and aggregation, rather than the accuracy of the underlying classification.
Key Findings: Frontier Models Struggle
The research reveals that even the most advanced frontier models, including GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro, struggle significantly with OOLONG. At a context length of 128K tokens, all evaluated models achieved less than 50% accuracy on both OOLONG-synth and OOLONG-real. Performance consistently dropped as the context window size increased, indicating a clear challenge in effectively processing and aggregating information over longer inputs.
Further analysis using OOLONG-synth showed that providing gold labels in the input resulted in only a small improvement in accuracy. This suggests that the primary bottleneck for models is not the individual classification of each piece of information, but rather the identification and aggregation of that information across the entire long context.
Temporal questions, which require reasoning about events before or after specific dates or across different time periods, proved to be the most challenging task type for models. This highlights a particular weakness in handling chronological information within long contexts.
Model-Specific Observations
The study also noted interesting behaviors from specific models. Gemini 2.5 Pro, while strong on OOLONG-real, experienced performance drops on OOLONG-synth due to frequently exceeding its maximum output length or triggering content filters. Deepseek R1, despite being a strong reasoning model, performed below a random baseline on OOLONG-synth, often failing to provide an answer or ending in incomplete sentences, suggesting difficulties in planning reasoning for information-dense tasks.
Also Read:
- CostBench: Unpacking How LLM Agents Plan and Adapt to Changing Costs
- Benchmarking LLMs for Cyber Threat Intelligence: Introducing AthenaBench
The Path Forward
The introduction of OOLONG provides a valuable and challenging benchmark for the AI community. The results clearly indicate that there is substantial room for improvement in designing LLMs that can robustly aggregate information and perform complex reasoning over large quantities of text. The data and evaluation harness for OOLONG are being released to foster further development in this crucial area of long-context understanding. You can find the full research paper here: OOLONG: Evaluating Long Context Reasoning and Aggregation Capabilities.
