TLDR: LongWeave is a new benchmark for evaluating Large Language Models (LLMs) in generating long, informative, and factual text. It introduces Constraint-Verifier Evaluation (CoV-Eval) to create realistic yet objectively verifiable tasks. The benchmark includes seven diverse tasks with customizable input/output lengths up to 64K/8K tokens. Evaluations on 23 LLMs show that even advanced models struggle with long-form generation, especially as complexity and length increase, highlighting common failure patterns and the need for better models.
Large Language Models (LLMs) have made significant strides in understanding and processing lengthy inputs. However, the ability to generate long, informative, and factually accurate outputs remains a considerable hurdle. Many existing benchmarks for long-form generation either rely on real-world queries with metrics that are difficult to verify or use synthetic setups that simplify evaluation but miss the nuances of real-world complexity.
Introducing LongWeave: A New Benchmark
To address this gap, researchers have introduced LongWeave, a novel benchmark designed to balance real-world relevance with objective verifiability. LongWeave employs a unique approach called Constraint-Verifier Evaluation (CoV-Eval). This method constructs tasks by first defining verifiable targets within real-world scenarios. It then systematically generates corresponding queries, textual materials, and constraints based on these targets. This ensures that the tasks are both realistic and can be objectively assessed, allowing for a rigorous evaluation of how well models meet complex real-world constraints.
LongWeave is highly flexible, supporting customizable input and output lengths, with inputs up to 64,000 tokens and outputs up to 8,000 tokens. It covers seven distinct tasks, each designed to challenge different aspects of long-form generation.
How CoV-Eval Works
The CoV-Eval mechanism reverses the traditional test construction process. Instead of extracting checklists from raw materials, which can be prone to errors, CoV-Eval starts with predefined, verifiable checklist objectives (Verifiers) that are grounded in real-world tasks. It then synthesizes corresponding inference queries, including Constraints and materials. The Constraint acts as a guided input that helps models generate the predefined Verifier, making verification and evaluation measurable. Each Constraint-Verifier (CV) pair in CoV-Eval maintains a deterministic one-to-one relationship under structurally defined rules, systematically linked to source materials.
The Seven Tasks of LongWeave
LongWeave evaluates LLMs across seven diverse tasks, each with varying Constraint-Verifier pairs:
- Code Fixing: Requires LLMs to fix Python code with style violations (e.g., Flake8) while ensuring the code remains runnable.
- KG to Text Biography Generation (BioG): Assesses the ability to generate coherent and factual biographies from knowledge graph triples.
- CSV Sales Report Analysis (SR): Evaluates LLMs in generating sales reports and answering specific questions based on a transaction table.
- AP Style News Writing (NW): Challenges models to write news articles adhering to the Associated Press Stylebook guidelines.
- KV Dictionary Generation (KVG): Tests the ability to generate a dictionary string with a target key-value pair at a specific index, following formatting rules.
- State Machine Simulation (SMS): Requires simulating step-by-step state transitions of a finite state machine.
- Paragraph Reordering (PR): Asks LLMs to reorder shuffled paragraphs into a coherent sequence.
Key Findings from Model Evaluations
An extensive evaluation of 23 LLMs on LongWeave revealed significant limitations in long-form generation. Even state-of-the-art models like DeepSeek-R1 reached a performance ceiling of 54.56%, with performance notably declining for 8,000-token outputs. Models often exhibit an input-output disconnect, meaning that while they can support inputs up to 64,000 tokens, they struggle to effectively synthesize this information into coherent, long-form responses. Surprisingly, simply expanding the input context window (e.g., to 1 million tokens) does not fundamentally solve long-generation challenges and can even degrade performance. Large-scale reasoning-oriented LLMs generally outperform general-purpose counterparts, but they sometimes fail to terminate their reasoning phase, leading to truncated outputs.
Common Failure Patterns
The research identified several common failure patterns:
- Instruction-following errors: Models often execute only the easy parts of a prompt, ignore harder constraints, or deviate from instructions as output length increases. Incomplete factual coverage and length control issues are also prevalent.
- Numerical errors: Frequent in quantitative tasks, where models miscalculate percentages, averages, or growth rates, suggesting a need for better tool support.
- Content problems: Include fabricated facts, redundancy, or looping, where models add unsupported information or repeat content.
- Reasoning-specific failures: Unique to reasoning models, where the reasoning phase doesn’t stop, leading to very long “thinking” traces and truncated final answers.
Also Read:
- Unpacking LLM Long-Context Abilities: Insights from the LooGLE v2 Benchmark
- Automated Peer Review for Large Language Model Evaluation
Conclusion
LongWeave provides a precise instrument to diagnose systemic issues in long-form generation and guide the development of truly capable models. The benchmark’s code is publicly available for further research and development. You can find more details about this research paper here: LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability.
