TLDR: ENTROPO is an entropy-enhanced preference optimization framework designed to improve Large Language Models (LLMs) for complex, multi-turn software engineering tasks. It addresses the problem of ‘diversity collapse’ in existing alignment methods by explicitly preserving the diversity of model outputs throughout multi-turn interactions. This allows for more effective ‘test-time scaling’ strategies, where models generate multiple diverse solutions and a hybrid selector picks the best one. ENTROPO achieves state-of-the-art results among open-weight models on SWE-bench benchmarks, making LLMs more robust and capable for real-world coding challenges by fostering exploration and preventing solutions from converging too narrowly.
Large Language Models (LLMs) have shown incredible potential across many fields, from understanding language to assisting with coding. However, when it comes to the intricate, multi-step challenges of software engineering, these advanced AI models often hit a wall. Tasks that require deep reasoning over vast codebases and coordinated use of various tools, like those found in benchmarks such as SWE-bench, remain particularly difficult for current LLMs.
One promising strategy to improve performance in these complex scenarios is called test-time scaling (TTS). This involves having the model generate multiple potential solutions and then selecting the best one. While effective, the success of TTS heavily relies on the diversity of the solutions the model can produce. If all generated solutions are too similar, the benefits of sampling more options diminish rapidly.
Existing methods for aligning LLMs with human preferences, such as Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO), are good at making models produce outputs that humans prefer. However, a side effect of this alignment process can be a reduction in the diversity of the model’s outputs, a phenomenon known as “diversity collapse.” This limits how much test-time scaling can actually help. Furthermore, most current preference optimization algorithms are designed for single-turn interactions, which don’t fully capture the complexity of multi-turn reasoning and tool integration needed for interactive coding agents.
To address these critical gaps, researchers have introduced a new framework called ENTROPO. This innovative approach enhances existing preference optimization algorithms to work effectively in multi-turn, tool-assisted environments. ENTROPO achieves this by explicitly adding an “entropy regularization” term to the preference objective. In simpler terms, it encourages the model to maintain a broader range of potential solutions, preventing diversity collapse. Crucially, ENTROPO extends this diversity-preserving objective from single-turn responses to entire multi-turn interactions, aligning the learning process with the sequential nature of complex coding tasks.
The ENTROPO framework also incorporates a hybrid best-trajectory selection scheme to maximize performance gains from test-time scaling. This scheme combines a learned verifier model, which scores potential solutions, with model-free approaches that favor high-quality trajectories (e.g., those that pass tests or involve a certain number of steps). This hybrid selector improves the effectiveness of sampling and amplifies the benefits of running multiple parallel solution attempts.
The effectiveness of ENTROPO was validated by fine-tuning a variety of models, ranging in size up to 106 billion parameters. The results are impressive: ENTROPO has set new state-of-the-art records among open-weight models on the SWE-bench leaderboard. For instance, a 30-billion-parameter model trained with ENTROPO achieved the top rank on SWE-bench-LITE and the fourth rank on SWE-bench-VERIFIED among open-weight models, surpassed only by models more than ten times its size. These findings underscore the vital role of preserving diversity for effective test-time scaling and establish ENTROPO as a robust method for developing powerful, interactive coding agents.
The research highlights that ENTROPO consistently outperforms standard DPO and KTO in test-time scaling settings, demonstrating that its entropy-preserving term is crucial for avoiding diversity collapse. Even smaller models, which initially showed negligible performance after standard fine-tuning, saw remarkable improvements with ENTROPO and test-time scaling, with resolve rates surpassing 10%. This indicates ENTROPO’s potential to make more efficient, smaller models viable for complex software engineering tasks.
Also Read:
- AI’s Adaptive Approach to Complex Optimization Challenges
- Collaborative AI Agents Enhance Prompt Optimization for Large Language Models
While the current implementation of ENTROPO focuses on offline preference learning, the core principle of entropy regularization could be extended to online reinforcement learning, potentially leading to even more robust policies. This work paves the way for developing more capable and reliable LLM-based tools that can tackle real-world software engineering challenges. You can read the full paper here: Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization.
