TLDR: A new research paper introduces an LLM-based Multi-Agent System (MAS) designed to evaluate Systematic Literature Reviews (SLRs) by automating protocol validation, methodological assessment, and topic relevance. The system, which adheres to PRISMA guidelines and uses 27 specialized agents, achieved an 84% agreement with human expert judgments across five diverse SLRs. It significantly reduces review time from weeks to minutes, offering a promising step towards scalable and accurate AI-driven support for interdisciplinary research workflows.
Systematic Literature Reviews (SLRs) are the backbone of evidence-based research, providing a structured way to identify, analyze, and synthesize existing work. However, the process of conducting and evaluating SLRs is notoriously labor-intensive, time-consuming, and can be prone to inconsistencies. With the explosion of scholarly publications, researchers face an increasing challenge in maintaining rigor and comprehensiveness.
A new research paper titled “Can Agents Judge Systematic Reviews Like Humans? Evaluating SLRs with LLM-based Multi-Agent System” explores a groundbreaking solution to these challenges. The paper introduces an innovative LLM-based Multi-Agent System (MAS) designed to act as an SLR evaluation co-pilot. This system aims to assist researchers in assessing the overall quality of systematic literature reviews by automating key steps like protocol validation, methodological assessment, and topic relevance checks.
Unlike traditional single-agent approaches, this new system employs a specialized agentic architecture that strictly adheres to the PRISMA guidelines – a widely recognized standard for reporting systematic reviews. This structured approach ensures more interpretable and reliable evaluations. The system is comprised of 27 specialized agents, organized into six PRISMA-aligned societies, each handling specific aspects of an SLR, such as the abstract, introduction, methods, results, discussion, and other information. Additionally, two utility agents manage PDF parsing and follow-up conversations.
When an SLR document is uploaded, an OCR-enabled Vision-Language Model converts it into structured text. A Coordinator Agent and Task Agent then break down the PRISMA checklist into modular evaluation tasks, dispatching them to the specialized agents. Each agent, powered by a state-of-the-art LLM (GPT-4.1), retrieves relevant research using an arXiv Toolkit, assigns a score from 0 to 5 for each PRISMA item, and provides qualitative feedback. A unique feature is the SLR-GPT Agent, which offers co-pilot style research support, suggesting new papers, verifying citations, cross-checking literature, and recommending editorial refinements to maximize PRISMA compliance.
To evaluate its effectiveness, the researchers conducted an initial study on five published SLRs from diverse fields, including Medical, E-commerce, AI, Metaverse, and IoT. The system’s PRISMA-based scores were compared against ratings provided by three expert SLR reviewers. The results were highly promising, showing an overall agreement of 84% between the system’s outputs and human expert judgments. The highest alignment was observed in the Introduction section (97%), followed by Discussion (94%) and Methods (93%). Even in sections with slightly lower agreement, such as Results (84%) and Other Information (81%), the alignment remained strong.
One of the most significant benefits highlighted by the study is the dramatic reduction in review time. While traditional peer review can take an average of 15 to 25 weeks, this automated MAS review system can analyze papers in just 15–20 minutes, depending on length and complexity. This offers early-stage insights that can significantly accelerate subsequent human reviews, thereby reducing the overall turnaround time for SLRs.
Also Read:
- Multi-Agent AI Framework Enhances Radiology Report Generation and Evaluation
- Steering AI Agents Towards Expert-Level Behavior
The paper acknowledges that while these early results are promising, the work represents a foundational step. Future directions include deploying an interactive, browser-based interface for users to engage with the system, revise summaries, and re-score sections. The researchers also plan to test the system with real-world users and integrate feedback to further align agent behavior with user preferences. Despite current limitations, such as evaluation on a small number of SLRs and integration primarily with arXiv, this study clearly demonstrates the feasibility of structured, agentic LLM support for SLRs, paving the way for more scalable and interactive systems in the future. You can read the full research paper here: Can Agents Judge Systematic Reviews Like Humans? Evaluating SLRs with LLM-based Multi-Agent System.
