TLDR: A new study investigates whether smaller and reasoning-focused Large Language Models (LLMs) can effectively score academic journal articles for research quality. It found that LLMs with over 4 billion parameters perform comparably to larger models. Averaging scores from multiple attempts consistently improved results, while few-shot prompting showed weak, inconsistent benefits. Reasoning models did not offer a clear advantage for this task, suggesting that smaller, non-reasoning models are practical for research evaluation, especially for ranking articles.
A recent study delves into the capabilities of various Large Language Models (LLMs), including smaller and reasoning-focused models, to assess the quality of academic journal articles. This research is particularly relevant given the increasing need for efficient and reliable methods to evaluate scholarly publications for departmental reviews, promotions, and grant applications.
Traditionally, research evaluation has relied on expert judgment, often supported by citation data. However, the emergence of LLMs offers a promising alternative. Previous studies have shown that large LLMs like ChatGPT and Gemini can produce quality scores that correlate well with expert opinions. This new paper expands on that by investigating whether smaller LLMs and those designed for reasoning can achieve similar performance, which is crucial for practical applications where larger models might be too slow or resource-intensive.
The Study’s Approach
The researchers addressed four key questions using a diverse set of LLMs, including Gemma3 variants, Llama4 Scout, Qwen3, Magistral Small, and DeepSeek R1. They tested these models on a dataset of 2,780 medical, health, and life science papers across six fields. To ensure robust evaluation, two different ‘gold standards’ were used: one based on aggregated departmental scores from the UK’s Research Excellence Framework (REF2021) and another with individual article scores assigned by the first author.
The study also explored two prompting strategies: ‘few-shot’ prompting, where the LLM is given a few examples of articles with their correct scores to guide its evaluation, and ‘averaging scores,’ which involves taking the mean of multiple scores generated from identical prompts. The goal was to see if these strategies could improve the LLMs’ ability to align with human expert judgments.
Key Findings
The results offer several important insights into the use of LLMs for research quality assessment:
-
Smaller LLMs Perform Well: The study found that smaller LLMs (specifically those with more than 4 billion parameters) demonstrated a substantial ability to score journal articles for research quality, performing similarly to larger cloud-based models like ChatGPT 4o-mini and Gemini 2.0 Flash. However, models with 1 billion parameters were often too small, and 4 billion parameters were sometimes insufficient, suggesting a minimum practical size for consistent performance across all fields.
-
Averaging Scores is Effective: A universally successful strategy was found to be averaging scores from multiple identical queries. This approach consistently improved the correlation between LLM scores and expert judgments across all tested contexts.
-
Few-shot Prompting Shows Mixed Results: While few-shot prompts (providing four examples) tended to help in some cases, the evidence was not conclusive. The improvement might be more related to the increased prompt variety rather than the LLM learning directly from the examples.
-
Reasoning Models Lack Clear Advantage: Surprisingly, reasoning models did not show a clear advantage over non-reasoning models for this specific task. They also tended to be significantly slower (5-10 times), which is a practical consideration against their use if they don’t offer superior performance.
Also Read:
- Beyond Size: Small Language Models Prove Competitive for Requirements Classification
- Boosting Code Quality with Compact AI Reward Models
Implications for Research Evaluation
These findings have significant implications for making LLM-based research evaluation more practical. Identifying smaller, effective models means that institutions with limited computing resources or strict data confidentiality requirements can still leverage AI for this task. The study highlights that LLMs are particularly valuable for ranking articles, helping to identify which papers are better than others or fall above/below certain quality thresholds, rather than providing perfectly accurate individual scores that match human scales.
For a deeper dive into the methodology and detailed results, you can read the full research paper here.
