TLDR: A new study evaluates how different prompting strategies (zero-shot, few-shot, chain-of-thought, self-reflection) interact with six large language models (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, Llama-4-Maverick) to automate systematic literature review (SLR) screening. It finds that CoT-few-shot offers the best precision-recall balance, while zero-shot maximizes recall. GPT-4o and DeepSeek show robust performance, and GPT-4o-mini offers competitive results at a lower cost. The research recommends a staged, cost-aware workflow for LLM deployment in SLRs, highlighting that optimal configurations depend on task complexity and model-prompt alignment.
Systematic Literature Reviews (SLRs) are a cornerstone of evidence-based research, providing transparent and reproducible syntheses of existing knowledge. However, the screening stage of SLRs, where researchers evaluate thousands of publications, is incredibly demanding and time-consuming. The rapid increase in scientific output has made manual screening increasingly difficult, limiting the scalability and timeliness of SLRs.
Recent advancements in artificial intelligence, particularly Large Language Models (LLMs), offer promising solutions to automate this labor-intensive screening process. LLMs excel at understanding natural language, reasoning, and classification, capabilities that align perfectly with the demands of literature screening. Early studies have already shown that LLMs can significantly reduce manual workloads while maintaining accuracy in abstract screening.
Exploring Prompting Strategies and LLMs
A new study, titled Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification, delves into how different prompting strategies interact with various LLMs to automate the screening stage of SLRs. The researchers evaluated six state-of-the-art LLMs: GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick. They tested these models under five different prompt types: zero-shot, few-shot, chain-of-thought (CoT), CoT-few-shot, and self-reflection. The evaluation focused on two main tasks: classifying the overall relevance of a study to SLR automation and classifying specific SLR tasks (searching, screening, retrieval, synthesis, writing) and the detection of LLM use.
Key Findings on Performance and Interaction
The study revealed significant interactions between the LLM used and the prompting strategy employed. The CoT-few-shot prompting strategy generally provided the most reliable balance between precision and recall, meaning it was good at identifying relevant studies without too many false positives. Zero-shot prompting, which provides no examples, maximized recall, making it ideal for initial passes where missing a relevant study is more critical than including a few irrelevant ones. Interestingly, self-reflection prompting, designed for iterative critique, performed poorly due to over-inclusivity and instability across models.
Among the LLMs, GPT-4o and DeepSeek-Chat-V3 consistently delivered robust overall performance. GPT-4o-mini also performed very competitively, especially considering its substantially lower cost. The research highlighted that no single prompting strategy or model was universally superior; instead, optimal performance often depended on tailoring the prompt to the specific model and task at hand.
Cost-Performance Considerations
A crucial aspect of deploying LLMs for large-scale SLRs is cost. The study included a cost-performance analysis for relevance classification, revealing substantial differences in cost among model-prompt pairings. GPT-4o-mini stood out as a low-cost option across all prompt types, offering attractive F1 scores (a balance of precision and recall) at a small incremental cost when used with structured prompts like CoT or CoT-few-shot.
Based on these findings, the researchers recommend a staged workflow for SLR automation. This approach involves deploying low-cost models with structured prompts for an initial, broad screening pass. Only borderline or contentious cases would then be escalated to higher-capacity, more expensive models for further adjudication. This strategy aims to preserve accuracy while significantly reducing overall expenditure.
Also Read:
- Adaptive Search: How Reinforcement Learning Powers Intelligent AI Agents
- Dissecting AI’s Approach to Optimization: A New Framework for Evaluating Language Models
Implications for SLR Automation
The study’s findings underscore the uneven but promising potential of LLMs to automate literature screening. While LLMs can already provide reliable support for tasks like screening classification, retrieval classification, and detecting LLM use, more abstract tasks such as synthesis and writing classification remain more challenging due to conceptual complexity and limited data. The research provides a valuable comparative benchmark and practical guidance for deploying LLMs in SLR workflows, emphasizing the importance of understanding model-prompt interactions and adapting strategies to specific task requirements and budget constraints.
