TLDR: A new study evaluates four types of reward models (discriminative/generative, outcome/process) for large language models across 14 diverse domains. Contrary to previous assumptions, generative outcome reward models (gORM) are found to be the most robust and effective in multi-domain settings. This is because process reward models (PRMs) struggle with long reasoning chains and are sensitive to label noise, challenging the idea that more granular, step-by-step supervision is always better for complex AI tasks.
Large Language Models (LLMs) are becoming increasingly powerful, but ensuring their reliability, especially during complex reasoning tasks, remains a significant challenge. A common approach to assess this reliability is through external verifiers or reward models. These models help distinguish correct reasoning from flawed logic, particularly when LLMs are scaled up for various applications.
Traditionally, it has been widely believed that Process Reward Models (PRMs), which evaluate every intermediate step of an LLM’s reasoning, are superior to Outcome Reward Models (ORMs), which only assess the final answer. This belief largely stemmed from observations in specific, often math-related, domains.
Challenging Conventional Wisdom Across Diverse Domains
A recent research paper, titled “Rethinking Reward Models for Multi-Domain Test-Time Scaling,” by Dong Bok Lee, Seanie Lee, Sangwoo Park, and their colleagues, presents a groundbreaking unified evaluation that challenges this long-held assumption. The researchers conducted the first comprehensive study of four key reward model variants: discriminative ORM (dORM), discriminative PRM (dPRM), generative ORM (gORM), and generative PRM (gPRM). Crucially, this evaluation spanned 14 diverse domains, moving beyond the narrow focus of previous studies.
The findings from this extensive evaluation were quite surprising and contradict the conventional wisdom:
-
Discriminative Outcome Reward Models (dORM) performed on par with discriminative Process Reward Models (dPRM).
-
Generative Process Reward Models (gPRM) were found to be largely uncompetitive.
-
Overall, the Generative Outcome Reward Model (gORM) emerged as the most robust and consistent performer, delivering significant and reliable improvements across every domain tested.
Why PRMs Struggle in Multi-Domain Settings
The researchers attribute the unexpected underperformance of PRMs in multi-domain settings to two primary factors:
1. The Risk Associated with Chain-of-Thought (CoT) Length
LLMs often produce longer reasoning chains, especially for more complex problems in diverse domains. PRMs, which aggregate scores step-by-step, tend to compound errors as the reasoning chain grows. This means that if an early step is misclassified as incorrect, the entire chain might be prematurely judged as flawed, even if the LLM later recovers from that error and arrives at a correct solution – a phenomenon the researchers refer to as an “aha” moment. The paper provides theoretical analysis and empirical evidence showing that the error rate of PRM variants increases linearly with the length of the CoT.
2. The Challenge of Label Noise
In specialized multi-domain datasets, obtaining high-quality, human-annotated process labels for every step of a reasoning chain is incredibly costly. As a result, many prior works rely on LLM-based auto-labeling, which can introduce noise into the process labels. The study found that dPRM is highly sensitive to this label noise, making it vulnerable in real-world multi-domain applications. While gPRM showed more robustness to label noise in math domains, it still suffered significantly in multi-domain settings due to a severe shift in its CoT-length distribution, caused by a process called “consensus filtering” during training.
Also Read:
- ContextPRM: Enhancing LLM Reasoning Across Diverse Fields by Focusing on Logical Flow
- Structural Reward Models: A New Approach to Interpretable and Efficient AI Evaluation
Practical Guidance for AI Developers
Based on their comprehensive analysis, the researchers offer practical guidelines for selecting the most appropriate reward model variant:
-
For short reasoning chains with clean labels and tight latency requirements, dPRM might still be suitable.
-
For long reasoning chains or scenarios with frequent error recoveries, gORM is recommended if computational resources permit; otherwise, dORM is a good alternative.
-
In mixed or shifting domains, gORM proves to be the most effective.
-
When dealing with high label noise, ORM variants are generally preferred as PRMs tend to amplify early errors.
-
For strict compute or latency constraints, dORM or dPRM are better choices, as generative models like gORM and gPRM introduce sampling overhead.
-
If training data is limited, gORM and gPRM demonstrate higher sample efficiency.
This research fundamentally shifts our understanding of how to best verify LLM reasoning, especially as these models are deployed in increasingly complex and diverse real-world applications. It highlights the critical role of generative outcome verification and provides a roadmap for future research in multi-domain settings. You can find more details about this study and access their code, datasets, and model checkpoints at the project’s GitHub repository.
