TLDR: A study reveals that Large Language Models (LLMs) used in academic peer review exhibit significant bias, consistently favoring authors from highly-ranked institutions. While explicit “hard ratings” might appear neutral, “soft ratings” expose deeper, hidden biases. Subtle gender preferences were also observed, with variations likely due to different model alignment strategies. These findings highlight critical fairness concerns as AI increasingly influences scientific evaluation.
Large Language Models (LLMs) are rapidly changing how academic papers are reviewed, from helping reviewers write detailed feedback to generating entire reviews automatically. While these AI tools offer exciting possibilities for efficiency, they also bring serious concerns about fairness and reliability. A recent study delves into these concerns, specifically investigating biases in LLM-generated peer reviews.
The research, titled “Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews,” conducted controlled experiments focusing on sensitive author metadata, such as institutional affiliation and gender. The findings consistently show a significant bias favoring institutions that are highly ranked on common academic lists. Additionally, the study found some subtle gender preferences, which, though small individually, could accumulate and worsen disparities over time. A key discovery was the presence of implicit biases, which became much clearer when looking at “token-based soft ratings.”
How the Study Uncovered Bias
To systematically evaluate bias, the researchers developed a controlled framework. They simulated a single-blind review setting, where reviewers (in this case, LLMs) are aware of author identities. For each paper, they generated review ratings using a standard prompt based on official review guidelines. To pinpoint sources of bias, they modified only one attribute at a time—either the author’s affiliation or their inferred gender—while keeping all other paper content constant. This method allowed for a clearer understanding of how changes in metadata influenced the LLM’s decisions.
The study introduced two types of ratings: “hard ratings” and “soft ratings.” Hard ratings represent the model’s most confident, explicit decision (an integer score). Soft ratings, however, capture the model’s internal confidence and uncertainty by looking at the probability distribution of its output. These soft ratings proved crucial for revealing more subtle and implicit forms of bias that might not be evident in the final, explicit scores.
Affiliation Bias: A Clear Preference for Prestige
The analysis of nine different LLMs revealed a consistent and strong preference for authors affiliated with high-status, or “Ranked-Stronger” (RS), institutions. This bias was evident in both explicit and hidden forms. For example, while a model’s hard rating might show only a small preference for an RS institution, its soft rating could reveal a much stronger underlying favoritism. This “hidden bias” suggests that even if models are trained to appear neutral in their final output, their internal scoring mechanisms might still be heavily skewed. This indicates a potential disconnect between the model’s internal reasoning and its publicly aligned behavior.
Some LLMs even explicitly mentioned affiliations in their reviews. For instance, one model occasionally flagged “Ranked-Weaker” (RW) affiliations as a concern, stating things like, “The affiliation is listed as University of Lagos, which raises a flag for potential resource constraints.” Conversely, RS affiliations were sometimes praised, with comments such as, “The authors are from CMU, so that’s a good sign.” These textual cues provide qualitative evidence of how models process and use author metadata in their evaluations.
Gender Bias: Subtle but Present
Gender-based biases were found to be less consistent across models compared to affiliation bias, but they were still present. Some models showed a tendency to assign higher hard ratings to male-associated names, while others favored female-associated names. This variation might stem from different “alignment strategies” used during model training, where developers aim to reduce social biases. However, this can sometimes lead to “overcompensation,” where models might favor perceived minority or underrepresented groups. The study highlights that these alignment policies can implicitly shape how gender is handled, even in contexts like peer review where identity should ideally be irrelevant.
Also Read:
- Unmasking AI: Evaluation Awareness Grows Predictably with Model Scale
- Detecting Hidden Agendas: How to Audit Ideological Bias in AI Chatbots
Why These Biases Matter
The systematic biases uncovered in LLM-generated reviews, particularly towards high-status institutions, raise serious concerns about fairness and reliability in academic evaluation. Even if final ratings seem neutral, the soft scores reveal deeper, implicit biases that post-training alignment might mask but not fully remove. This discrepancy between internal preferences and surface-level outputs is critical, especially as LLM-assisted systems increasingly influence high-stakes tasks like deep research and the shaping of scientific literature. Even small, systematic biases can have significant long-term consequences when scaled across many review cycles and academic careers.
The paper concludes by emphasizing the importance of rigorous evaluation and the complexity of aligning LLMs for equitable decision-making in critical tasks like paper reviewing. It serves as a crucial reminder that while AI offers powerful tools, their integration into sensitive processes requires careful consideration of their inherent biases. You can read the full research paper for more details at arxiv.org/pdf/2509.13400.
