TLDR: This research paper explores the effectiveness of smaller language models, specifically the Phi-4 family, as reward models for improving code generation by Large Language Models (LLMs). By fine-tuning these models to evaluate both final code outcomes and intermediate steps, the study demonstrates that they can act as efficient ‘code critics’. The 14B Phi-4 model, in particular, achieved over a 20% improvement in selecting correct Python code solutions, proving that significant model size is not always necessary for impactful reward modeling in this domain.
Generating high-quality code with Large Language Models (LLMs) is a significant challenge in artificial intelligence. To help LLMs improve their reasoning abilities in this area, reward models play a crucial intermediate role. These models are designed to evaluate either the final outcome of a generated code snippet or the intermediate steps taken during its creation.
Traditionally, it’s believed that a model’s reflection capabilities increase with its size. However, this research paper, titled “Smaller Models, Smarter Rewards: A Two-Sided Approach to Process and Outcome Rewards”, investigates whether smaller, state-of-the-art language models, specifically the Phi-4 family, can be effectively transformed into usable reward models. The goal is to blend the consideration of both process rewards (evaluating intermediate steps) and outcome rewards (evaluating the final result).
The authors, Jan Niklas Groeneveld, Xi Qin, Alexander Schaefer, and Yaad Oren, constructed a unique dataset of code samples. These samples were labeled for correctness using the APPS coding challenge benchmark. They then trained a special type of model, a value-head model, to estimate the probability of success for intermediate outputs. This involved taking existing decoder-only transformer models, like the Phi-4-mini (3.8B parameters) and Phi-4 (14B parameters), and adding a regression layer, followed by supervised fine-tuning.
Reward models typically fall into two categories: Process Reward Models (PRMs), which score intermediate steps, and Outcome Reward Models (ORMs), which score only the final output. ORMs are simpler and cheaper but can suffer from sparse feedback, making it harder to pinpoint exactly where a mistake occurred. PRMs offer more detailed, step-by-step feedback, improving learning efficiency, but they often require expensive human annotations for intermediate steps during training.
This paper aimed to build a versatile reward model capable of serving both roles. The evaluation showed promising results: these smaller LLMs are indeed capable of acting as effective reward models or ‘code evaluation critics’. They successfully identified correct solutions among multiple candidates. Using this critic, the researchers observed a significant improvement of over 20% in the ability to find the most accurate code from multiple generated options.
The study also delved into how these models perform when judging intermediate generation steps. While the models needed a ‘warming-up’ stage, they began to show meaningful performance (better than random guesses) as PRMs after approximately 50% of the token generation process. This indicates they can catch errors in the code, even if they might lack the self-correction capability for already incorrect reasoning steps early on.
In conclusion, the research demonstrates that large-scale capacity is not always a prerequisite for effective reward modeling. Models from the Phi-4 family, particularly the 14B parameter version, can serve as powerful reward models, capable of identifying errors and substantially increasing the probability of selecting correct code. However, model size still plays a role, as the 3.8B Phi-4-mini-instruct model performed less effectively than its 14B counterpart.
Also Read:
- Unlocking Advanced Reasoning in Language Models with Code Execution
- Enhancing Mathematical Reasoning in Language Models: A Reinforcement Learning Approach to Budget Forcing
This work opens doors for more efficient and accessible methods to improve code generation quality in LLMs, balancing performance with computational resources.
