TLDR: Generative Self-Refinement (GSR) is a novel method that enables a single Large Language Model (LLM) to generate multiple candidate solutions for complex problems and then critically refine them itself. Unlike traditional methods that are limited by the quality of initial candidates, GSR, supported by a unique hybrid training approach, can produce correct answers even when all initial attempts are flawed. This framework achieves state-of-the-art performance on mathematical benchmarks and demonstrates robust generalization across different model scales and to unseen reasoning tasks.
Large Language Models (LLMs) have made incredible strides towards Artificial General Intelligence, largely due to advancements in their training. However, a complementary approach called test-time scaling (TTS) has also gained attention, focusing on improving model performance by allocating additional computing power during inference.
A significant challenge for LLMs is solving complex, multi-step reasoning problems. Current TTS methods, such as majority voting (picking the most frequent answer) or Best-of-N (using an external verifier to select the best response), have limitations. Their effectiveness is tied to the quality of the initial candidate responses. If all candidates are incorrect, these methods cannot produce a correct solution. Furthermore, introducing an additional model for selection or fusion adds to deployment costs and complexity.
To address these issues, researchers from Microsoft and Peking University have introduced a novel framework called Generative Self-Refinement (GSR). This approach empowers a single, unified LLM to generate a set of candidate responses in parallel and then perform self-refinement to synthesize a superior solution. The model is prompted with the original problem and its own generated candidates, allowing it to leverage its intrinsic reasoning ability to diagnose flaws and selectively synthesize valid insights.
How Generative Self-Refinement Works
The GSR process involves two main stages. First, the LLM generates several diverse candidate solutions. These raw outputs are then parsed, and summary content is extracted. In the second stage, these candidates are used to construct an augmented prompt, which includes the original problem and the candidate solutions. The model is explicitly instructed to analyze the connection between the candidates and the problem, and to selectively use valuable insights. Crucially, even if all initial candidates are flawed, the model is required to reason independently and produce a correct final solution. This means GSR can achieve a significantly higher performance boundary than methods like majority voting or Best-of-N, as it’s not limited by the quality of the initial candidate pool.
Hybrid Training for Dual Abilities
While the concept of self-refinement is powerful, simply prompting an LLM to do it doesn’t always yield optimal performance. This is because data specifically for refining responses is often scarce in standard training datasets. The researchers found a synergistic relationship between generating effective responses and integrating multiple responses into a better answer. Therefore, they designed a hybrid training pipeline that jointly optimizes for two complementary objectives: direct problem-solving and self-refinement. This pipeline equips the model with the dual abilities to generate high-quality responses directly and to self-refine existing candidates.
Also Read:
- Unveiling the Silent Thought Processes of Large Language Models
- Ensuring Language Models Reason for the Right Reasons
Impressive Results and Generalization
Extensive experiments on five challenging mathematical benchmarks demonstrated that GSR achieves state-of-the-art performance. For instance, on the AIME24 benchmark, the GSR-7B model’s pass@1 accuracy dramatically increased from 13.2% to 50.1%. More remarkably, a fine-grained analysis showed that even when all initial candidate solutions were incorrect, GSR could still produce a correct solution a significant percentage of the time (e.g., 5.9% on AIME24), a feat impossible for baseline methods like majority voting or Best-of-N.
The learned self-refinement skill proved to be robust and generalizable. It was effective across different model scales (7B, 14B, and 32B models) and even generalized to out-of-distribution reasoning tasks, such as Knights and Knaves logic puzzles, despite being trained exclusively on math datasets. This suggests that the training fosters a general, model-agnostic ability to evaluate proposed solutions regardless of their origin and aggregate a superior one.
This research highlights a promising direction for developing more capable and efficient LLM reasoners through a self-contained refinement framework. For more details, you can read the full research paper here.
