TLDR: AcademicEval is a new live benchmark for evaluating Large Language Models (LLMs) on long-context academic writing tasks (Title, Abstract, Introduction, Related Work). It uses arXiv papers for data, eliminating manual labeling and preventing label leakage through periodic updates. The benchmark features flexible context lengths via co-author graph-based few-shot demonstrations. Initial evaluations show LLMs struggle with hierarchical abstraction and long demonstrations, highlighting challenges in long-context modeling and revealing nuanced performance differences between LLMs and Retrieval-Augmented Language Models.
Large Language Models (LLMs) have shown impressive capabilities in understanding long texts. However, evaluating these models, especially their ability to handle extensive contexts, has been a challenge. Existing benchmarks often suffer from limitations such as fixed context lengths, the need for extensive manual labeling, and the risk of “label leakage,” where the models might have already seen the test data during their training.
To address these issues, researchers from the University of Illinois at Urbana-Champaign have introduced a new benchmark called AcademicEval. This innovative platform is designed for evaluating LLMs on long-context generation tasks, specifically focusing on academic writing. AcademicEval stands out because it uses real papers from arXiv, eliminating the need for manual labeling and ensuring high-quality, expert-curated data.
AcademicEval features four distinct academic writing tasks: Title, Abstract, Introduction, and Related Work generation. These tasks cover a range of abstraction levels, meaning some require a very high-level summary (like a title), while others demand more detailed and structured content (like an introduction or related work section). A key aspect of AcademicEval is its flexible context length, which is achieved by integrating few-shot demonstrations. These demonstrations are drawn from a collected co-author graph, providing relevant and high-quality examples to the LLMs.
One of the most significant features of AcademicEval is its “live evaluation” mechanism. By periodically updating the benchmark with the latest papers from arXiv, it effectively prevents label leakage. This means that the evaluation data is always fresh and unlikely to have been part of the LLMs’ training datasets, leading to a fairer and more accurate assessment of their true capabilities.
The researchers conducted a comprehensive evaluation using AcademicEval, testing various LLMs, including standard models, long-context LLMs, and Retrieval-Augmented Language Models (RALMs). The results revealed that LLMs generally struggle with tasks requiring hierarchical abstraction levels and tend to perform poorly with very long few-shot demonstrations. This highlights the challenging nature of the benchmark and points to areas where LLMs need improvement in long-context modeling.
Interestingly, the evaluation showed that RALMs often achieved the strongest results in automatic metrics like BERTScore and ROUGE-L. This is likely because retrieval methods can concentrate relevant information into shorter, more manageable chunks. However, an “LLM-as-a-Judge” evaluation, which assessed qualities like novelty, feasibility, consistency, factuality, and academic style, presented a more nuanced picture. For tasks like Title and Abstract generation, retrieval was not always preferred, while it proved highly beneficial for Related Work generation.
The study also explored the impact of context length, finding that performance often degraded as the input length increased. This further emphasizes the challenge LLMs face in effectively processing ultra-long inputs. The integration of few-shot demonstrations had mixed effects, suggesting that current LLMs don’t always fully leverage long in-context learning effectively. However, demonstrations from co-author papers generally had a more positive impact than randomly selected ones.
Also Read:
- Enhancing Scientific Discovery with AI-Powered Research Idea Evaluation
- Adaptive Search: How Reinforcement Learning Powers Intelligent AI Agents
AcademicEval offers valuable insights for enhancing LLMs’ long-context modeling capabilities and provides a robust framework for future research in this area. For more details, you can read the full research paper available at arXiv.
