TLDR: This research introduces an uncertainty-driven framework for efficiently creating high-quality training data for Process Reward Models (PRMs), which evaluate intermediate steps in LLM mathematical reasoning. It also proposes two new output aggregation methods, Hybrid Majority Reward Vote and Weighted Reward-Frequency Vote, that combine the strengths of majority voting and PRMs to improve answer selection. Experiments show the framework’s efficiency and effectiveness in boosting LLM performance on math tasks.
Large language models (LLMs) have shown impressive abilities in solving complex math problems. However, they often make mistakes in the intermediate steps of multi-step solutions. To address this, Process-level Reward Models (PRMs) were developed. PRMs evaluate each step of a solution, providing detailed feedback that helps improve the LLMs’ reasoning skills.
A significant challenge in building effective PRMs is the need for high-quality data that labels the correctness of each step. Traditional methods for creating this data, such as human annotation, are very expensive and time-consuming. Even automated methods, like those using Monte Carlo techniques or strong LLMs as judges, can be computationally intensive and sometimes inefficient.
A New Approach to Data Construction
Researchers Jiuzhou Han, Wray Buntine, and Ehsan Shareghi have introduced an innovative framework that uses uncertainty to automate the creation of process reward data. Their method, detailed in their paper “Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning,” focuses on two main areas: generating diverse candidate solutions and efficiently annotating them.
First, for data generation, their framework samples multiple solutions from an LLM for a given math problem. It then calculates an “uncertainty score” for each solution based on the model’s confidence in generating each token. The framework specifically selects the most uncertain correct and incorrect solutions. This targeted selection ensures that the training data for PRMs includes challenging examples, helping the PRM learn to identify subtle errors more effectively.
Second, for data annotation, the framework uses an uncertainty-driven automated process. For correct solutions, all steps are labeled as true. For incorrect solutions, the method identifies the step where the model’s uncertainty increases the most, as this is likely where an error occurred. This approach significantly reduces the computational cost of annotation while maintaining high quality, making the process more scalable than previous methods.
Improving Answer Selection with Hybrid Aggregation
Beyond data construction, the paper also addresses limitations in how LLMs’ multiple outputs are combined to get a final answer. Common methods like “Majority Vote” (selecting the most frequent answer) can fail if answers are too varied or if the model is confidently wrong. PRMs, while good at step-level evaluation, might struggle with unusual or difficult problems.
To overcome these issues, the researchers propose two new uncertainty-aware output aggregation methods: Hybrid Majority Reward (HMR) Vote and Weighted Reward-Frequency (WRF) Vote. These methods combine the strengths of both Majority Vote and PRMs.
The HMR Vote first checks if a clear majority answer exists. If not, it uses the PRM to select the solution with the highest overall reward (minimum step score). The WRF Vote, on the other hand, calculates a combined score for each unique answer, considering both its frequency among sampled solutions and its average PRM reward. The answer with the highest combined score is then chosen. The paper notes that WRF generally proves to be the more robust strategy.
Also Read:
- RL-PLUS: A New Approach to Expand LLM Reasoning Capabilities Beyond Current Limits
- How Causal Graphs Guide Large Language Models in Complex Mathematics
Experimental Validation and Impact
The effectiveness and efficiency of this uncertainty-driven framework were tested extensively on datasets like ProcessBench, MATH, and GSMPlus. The results show that PRMs trained with the uncertainty-driven data (UnPRM40K) consistently perform better than those trained with similarity-driven data. The uncertainty-driven annotation method also proved to be significantly more computationally efficient, reducing verified steps and sampled tokens compared to other methods.
Furthermore, the HMR and WRF aggregation strategies consistently improved performance across different PRMs and mathematical reasoning tasks, often outperforming standard Majority Vote and traditional PRM methods. This indicates that integrating both the consensus among answers and the fine-grained step-level evaluation from PRMs leads to more reliable answer selection.
This research provides a valuable framework for building better PRMs and improving the accuracy of LLMs in complex mathematical reasoning. The code and data for this work are planned to be publicly available, which you can find more about by reading the full paper here: Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning.
