TLDR: EntropyLong is a novel data construction method for training long-context language models. It leverages a model’s predictive uncertainty (entropy) to identify information deficits in documents. The method then retrieves semantically relevant contexts and empirically verifies if these contexts reduce the model’s uncertainty. This ‘model-in-the-loop’ verification ensures that only genuinely useful long-range dependencies are included in training data. This approach leads to significant performance improvements on long-context benchmarks like RULER and LongBench-v2, and effectively mitigates the ‘lost-in-the-middle’ problem.
Large Language Models (LLMs) are becoming increasingly powerful, capable of processing and reasoning over vast amounts of information. However, a significant challenge remains: effectively training these models to utilize extremely long contexts. While architectural advancements have expanded theoretical context windows to millions of tokens, there’s a persistent bottleneck – a scarcity of training data that genuinely enables models to leverage this extended context. Current methods often rely on simply concatenating short, unrelated documents or using heuristic-based approaches, which frequently fail to create meaningful long-range dependencies.
A new research paper, titled “EntropyLong: Effective Long-Context Training via Predictive Uncertainty,” introduces a novel solution to this problem. Authored by Junlong Jia, Ziyang Chen, Xing Wu, Chaochen Gao, Zijia Lin, Debing Zhang, Songlin Hu, and Binghui Guo, the paper proposes a data construction method called EntropyLong. This approach moves beyond traditional heuristics by using a model’s own predictive uncertainty to verify the quality of dependencies within long contexts. You can read the full paper here.
The Core Idea: Predictive Uncertainty as a Guide
EntropyLong’s central insight is that a model’s predictive uncertainty, specifically its entropy, directly signals an information deficit. When a model struggles to predict the next token (a high-entropy position), it indicates a point where additional context could be maximally beneficial. The method then retrieves semantically relevant contexts and verifies their utility by assessing whether they reduce this prediction entropy. This “model-in-the-loop verification” ensures that each dependency introduced represents a measurable information gain, rather than just a spurious correlation.
How EntropyLong Works: A Four-Stage Pipeline
The EntropyLong framework operates through a four-stage pipeline:
1. Adaptive Threshold-Based High-Entropy Position Selection: The system identifies positions in a document where the language model is most uncertain about its next prediction. This is done by calculating the predictive entropy at each token and selecting those that exceed an adaptive threshold based on the document’s overall entropy distribution.
2. Information-Theoretic Context Retrieval: For each identified high-uncertainty position, the surrounding text is used as a query to retrieve potentially relevant document chunks from a large corpus. This step aims to find information that could resolve the model’s uncertainty.
3. Entropy Reduction Verification: This is the crucial verification step. Each retrieved candidate context is prepended to the original document, and the model’s entropy at the high-uncertainty position is re-evaluated. Only contexts that demonstrably reduce the model’s predictive uncertainty by a certain threshold are retained. This ensures that the added context is genuinely helpful.
4. Strategic Concatenation with Root Document: The original document is then combined with these verified contextual supplements to create training sequences with validated long-range dependencies. The researchers found that randomly shuffling the retrieved contexts before concatenation yielded better performance.
Building a High-Quality Dataset and Achieving Superior Performance
Using this methodology, the researchers constructed a comprehensive 128K-length training corpus from FineWeb-Edu and Cosmopedia, containing samples with verified long-range dependencies. Models trained on this EntropyLong dataset demonstrated significant improvements on RULER benchmarks, especially in tasks requiring distant information, across context lengths from 8K to 128K. For instance, on 128K context length, EntropyLong achieved a score of 81.26, significantly outperforming Quest (60.11) and NExtLong (77.99).
Furthermore, after instruction fine-tuning, EntropyLong models achieved substantial gains on LongBench-v2, showcasing enhanced long-context understanding in real-world scenarios. The benefits were particularly evident in “Long” tasks, where EntropyLong showed an 8.40-point improvement over the best baseline.
Addressing the “Lost-in-the-Middle” Problem
A common issue with long-context models is the “lost-in-the-middle” phenomenon, where models struggle to retrieve information located in the middle of a long sequence. EntropyLong’s approach helps mitigate this. Analysis of attention patterns revealed that EntropyLong consistently maintains higher attention to correct answers, regardless of their position within the sequence, and shows significant relative improvements over baselines in middle positions (25%-75% distance), directly validating its ability to overcome this limitation.
Also Read:
- ChunkLLM: A New Approach to Faster and More Efficient Large Language Model Inference
- How Language Models Learn to Balance Internal Knowledge with New Information
Conclusion
EntropyLong represents a significant step forward in training long-context language models. By shifting from heuristic-based data synthesis to an evidence-based approach grounded in information-theoretic principles, it ensures that training data contains genuine, measurable long-range dependencies. This leads to more capable and robust LLMs that can effectively process and reason over extensive information spans, paving the way for more automated and efficient data curation pipelines.
