TLDR: A new research paper introduces AllSummedUp, an open-source framework to compare text summarization evaluation metrics, from classical ROUGE to LLM-based methods like G-Eval and SEval-Ex. The study reveals significant discrepancies in reproducibility between reported and observed metric performances, highlighting that metrics with high human alignment are often computationally expensive and less stable. It also shows vast differences in execution times, with LLM-based metrics taking hours compared to seconds for simpler ones. The framework advocates for standardized protocols and transparent reporting to improve reliability in automatic summarization assessment.
In the rapidly evolving field of Natural Language Processing (NLP), particularly in automatic text summarization, the ability to reliably evaluate the quality of generated summaries is crucial. However, a new research paper titled “AllSummedUp: an open-source framework for comparing summary evaluation metrics” by Tanguy Herserant and Vincent Guigue sheds light on significant challenges related to the reproducibility and consistency of these evaluation metrics. This study introduces an open-source framework designed to bring transparency and fairness to the comparison of various summarization evaluation metrics, from traditional methods to advanced Large Language Model (LLM)-based approaches.
The core problem addressed by the researchers is the growing concern about reproducibility in scientific research, especially within NLP. With the rise of LLMs, which are increasingly used to evaluate or generate text, a worrying variability in their outputs has been observed, even under seemingly deterministic conditions. This instability directly impacts the reliability of evaluation metrics, which are vital given the high cost of human judgment in summarization tasks. The paper highlights how even minor changes, like adding a space to a prompt, can significantly alter LLM outputs, a phenomenon dubbed the “Butterfly Effect.” Furthermore, the reliance on specific, sometimes proprietary, models poses a long-term reproducibility risk if these models become inaccessible or are modified.
The AllSummedUp framework aims to tackle these issues head-on. It is a unified, open-source platform that rigorously tests the reproducibility of results across six representative evaluation metrics. These metrics span a range of approaches, including classical lexical overlap metrics like ROUGE, and more recent LLM-based methods such as G-Eval, SEval-Ex, QuestEval, UniEval, BARTScore, and BERTScore. The framework is applied to the widely used SummEval dataset, which contains 1600 summaries generated by 16 automatic systems, along with human annotations across four dimensions: coherence, factual consistency, fluency, and relevance.
The framework is structured with three main components: a standardized Metric Interface for consistent implementation, a central Evaluator to manage and apply metrics, and a Report Generator to format results and create visualizations. This modular design ensures that all metrics follow the same workflow, producing comparable outputs and reducing implementation-related variability. It is also designed to be extensible, allowing for the easy addition of new evaluation datasets.
The experiments conducted using the AllSummedUp framework revealed several critical findings. Firstly, there were notable discrepancies between the performance reported in existing literature and the results observed within the experimental setting. For instance, some metrics showed significant deviations or even inversions of trends compared to their reference values. This suggests that reported performances are not always generalizable, even when using the same dataset. The researchers attributed these differences to factors such as using different LLM versions, modified prompts, or varying software library versions. A specific challenge was noted with ROUGE, where a lack of detailed methodology in original papers prevented exact replication of results.
Secondly, the study highlighted a stark contrast in computational costs. “Light” metrics like ROUGE and BARTScore executed in mere seconds, making them suitable for rapid or large-scale evaluations. In contrast, “heavy” LLM-based metrics such as G-Eval, SEval-Ex, and QuestEval required significantly more processing time, often several hours for a single run. The size of the underlying LLM also played a major role; for example, Qwen-2.5-72b consistently took three to four times longer than Gemma-3-27b for identical metrics. This underscores the practical limitations of using computationally intensive LLM-based metrics without dedicated infrastructure.
These findings reveal a structural trade-off: metrics that align best with human judgments tend to be the most computationally expensive and less stable across different runs. While LLM-based metrics can offer higher correlations, their high computational cost and potential for variability pose significant challenges. Intermediate approaches like UniEval, however, offered a good compromise, providing relatively stable results across multiple dimensions with moderate execution times.
The study also pointed out several limitations, including the partial reproducibility of some original metrics, restricted access to proprietary models (like GPT-4 for G-Eval), and the reliance on a single benchmark dataset (SummEval). The prohibitive computational time for certain metrics also limits their applicability in resource-constrained environments.
Also Read:
- AI Agent Memory: Why Simple Context Management Outperforms Complex Summarization
- MENTAT: A Hybrid AI Method for Precise Numerical Predictions in Complex Reasoning Tasks
In conclusion, the AllSummedUp framework not only exposes the major challenges in automatic summarization evaluation but also offers a valuable open-source tool for the community. The research emphasizes the urgent need for more rigorous, explicit, and systematic approaches, advocating for standardized protocols, comprehensive publication of experimental parameters (models, prompts, temperatures, random seeds), and the integration of statistical measures of variability in performance reports. These efforts are crucial for fostering a better understanding of existing tools and for developing new metrics that balance methodological rigor, empirical robustness, and computational efficiency in the age of LLMs. You can find the full research paper here.
