TLDR: DREAM (Dual-phase REward-guided Adaptive reasoning framework at test tiMe) is a new method that improves the efficiency and accuracy of Large Language Models (LLMs) in complex reasoning tasks. It achieves this by explicitly separating the reasoning process into distinct planning and execution phases, each guided by its own reward model. Additionally, DREAM features a dynamic budget allocation mechanism that adaptively distributes computational effort, allowing for early stopping on easy steps and reallocating resources to more challenging ones. Experiments on math reasoning and code generation benchmarks show that DREAM consistently outperforms prior methods, demonstrating better accuracy and efficiency, along with strong generalization capabilities.
Large Language Models (LLMs) have made remarkable strides in tackling complex reasoning tasks, from solving mathematical problems to generating code. A common strategy to enhance their performance involves using tree-based search methods, where LLMs explore various reasoning paths, guided by ‘reward models’ that help prune less promising options.
However, current methods, while effective, often fall short in efficiency. They tend to treat the entire reasoning process as a single unit, failing to distinguish between the high-level ‘planning’ and the detailed ‘execution’ phases inherent in many tasks. This can lead to wasted computational effort. For instance, if a plan is good but the execution is flawed, the entire path might be discarded, losing the value of the good plan. Conversely, if a plan is poor, the system might still spend resources generating its execution.
Another limitation is the rigid allocation of computational budget. Many existing systems assign a fixed amount of effort to each step, regardless of its difficulty. This means easy steps might receive excessive attention, while more challenging parts of a problem remain underexplored.
To address these challenges, researchers have introduced a novel framework called DREAM: Dual-phase REward-guided Adaptive reasoning framework at test tiMe. This innovative approach explicitly separates the reasoning process into two distinct phases: planning and execution. For more details, you can read the full paper here.
How DREAM Works
DREAM operates by conducting a search over planning candidates first. It uses a dedicated ‘planning reward model’ to select the most promising subgoals. Once a plan is chosen, it then searches over execution candidates, applying a separate ‘execution reward model’ to identify the most reliable solutions. This two-stage process ensures that inefficient plans are discarded early, and good plans can be paired with multiple execution attempts until a correct result is found, leading to a more efficient allocation of computation.
Beyond the dual-phase search, DREAM also incorporates a dynamic budget allocation mechanism. This feature adaptively adjusts the number of samples taken in both the planning and execution phases based on real-time feedback from the reward models. If confident candidates are found early for a simple step, sampling can be terminated, saving resources. Conversely, if a step proves particularly challenging, additional computational budget can be allocated to explore it more thoroughly. This adaptive strategy aligns computational effort with the actual difficulty of each step, improving overall accuracy and efficiency.
Experimental Validation
The effectiveness of DREAM was rigorously evaluated across two major domains: mathematical reasoning (using benchmarks like GSM8K and MATH) and code generation (using HumanEval and MBPP). The results consistently demonstrated that DREAM not only improved answer accuracy but also enhanced test-time efficiency compared to standard beam search and other reward-model-based methods.
A significant finding was the strong generalization ability of DREAM’s reward models. Even when tested on datasets not used during their training, the models provided effective guidance, indicating their robustness and transferability across different tasks and LLM backbones. The research also highlighted a synergy between the dual-phase search and dynamic budget allocation, where both components mutually reinforce each other to reduce wasted computation and reallocate resources effectively.
Also Read:
- PRISM: A Dynamic Strategy Framework for Enhanced Mathematical Reasoning in LLMs
- Chain-in-Tree: Streamlining LLM Reasoning with Adaptive Branching
Conclusion
DREAM represents a significant advancement in test-time scaling for LLMs. By explicitly separating planning and execution, guiding each phase with reward models, and adaptively managing computational budget, it offers a more fine-grained and efficient approach to complex reasoning tasks. This framework holds promise for improving the performance of LLMs in a wide range of applications requiring multi-step reasoning.
