TLDR: Researchers developed Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a new post-training method that improves LLMs’ decision-making abilities. It works by having LLMs generate decision trajectories, selecting the lowest-regret ones, and then fine-tuning the model on these high-performing examples. This approach, applied to various LLMs (Transformers, open-weight, closed-weight), consistently reduced regret, improved exploration-exploitation balance, and showed strong generalization across diverse tasks and contexts, even enabling models to discover decision-making algorithms autonomously.
Large language models (LLMs) are increasingly being used as “agents” to help with decision-making in various interactive and changing environments. However, these models weren’t originally designed for such tasks, and recent research has shown they can struggle with even basic online decision-making problems. They often fail to achieve low “regret” – a measure of how much worse an agent’s decisions are compared to the best possible decisions in hindsight – or to effectively balance exploration (trying new things) and exploitation (using what works best).
To tackle this challenge, researchers from the Massachusetts Institute of Technology and the University of Maryland, College Park, have introduced a new post-training method called Iterative Regret-Minimization Fine-Tuning (Iterative RMFT). This innovative procedure repeatedly refines LLMs by distilling low-regret decision trajectories back into the base model. The core idea is to learn from the model’s own best-performing behaviors.
How Iterative RMFT Works
At each iteration of Iterative RMFT, the LLM generates several decision trajectories for a given online decision-making task. These trajectories are essentially sequences of decisions and actions taken by the model. The procedure then identifies and selects the “k” trajectories that resulted in the lowest regret. These top-performing trajectories are then used to fine-tune the model through a process called supervised fine-tuning. This iterative loop allows the model to continuously improve its decision-making capabilities.
Unlike previous methods that might rely on distilling actions from known algorithms or using rigid, manually crafted reasoning structures, Iterative RMFT uses the regret metric to automatically identify and enhance the model’s decision-making ability. Crucially, it integrates the model’s own self-generated reasoning rationales. This flexibility in handling output and reasoning formats is a significant advantage, providing more adaptable training signals in natural language.
Broad Applicability and Key Findings
The researchers applied Iterative RMFT across a spectrum of models and decision-making environments. They started with Transformers that handle numerical inputs and outputs, then moved to lightweight open-weight LLMs like Phi-3.5-mini-instruct, Gemma-2-9b-it, and Qwen3-8B, and finally to more advanced closed-weight LLMs such as GPT-4o mini. The decision-making environments included Full-Information Online Learning (FOL), Multi-Armed Bandits (MABs), and Non-Stationary Multi-Armed Bandits (NS-MABs).
The empirical results were consistently positive. Iterative RMFT significantly improved LLMs’ decision-making performance. The trained models exhibited lower regret values, a better balance between exploration and exploitation, and impressive generalization capabilities. This means they could perform well on tasks with varying time horizons, different action space sizes, new reward generation processes, and diverse decision-making contexts described in natural language.
For instance, in Full-Information Online Learning, the trained Transformer models converged empirically to known online learning algorithms like Follow-the-Regularized-Leader (FTRL) and the Hedge algorithm, demonstrating a theoretical foundation for the approach. For open-weight LLMs, the method led to lower regret and improved exploration-exploitation tradeoffs, even generalizing to different reward types and longer time horizons. When applied to GPT-4o mini in complex, real-world language-grounded scenarios, the trained model showed enhanced robustness, better semantic-numerical alignment, and a more effective exploration-exploitation strategy compared to its base version.
A notable aspect is the flexibility Iterative RMFT offers regarding output and reasoning formats. This allows the trained models to naturally generalize across tasks that differ in time horizon, action space size, reward generation processes, and decision-making contexts described by natural language. The approach essentially empowers LLMs to autonomously discover algorithms for decision-making, guided by the regret metric.
Also Read:
- Tool Zero: A New Path for LLMs to Master Tools Through Pure Reinforcement Learning
- AI Learns to Race: Automated Reward Design for Gran Turismo 7
Theoretical Insights and Future Directions
The paper also provides theoretical insight into how a single-layer attention Transformer model, under this training paradigm, can lead to a “no-regret” learner in a simplified setting. This suggests that no-regret behaviors can naturally emerge through this iterative self-imitation process.
While promising, the study acknowledges certain limitations, such as training being limited to relatively short horizons due to computational budget and context-length constraints. The tasks were mostly synthetic or procedurally generated. Future work includes training on genuinely long-horizon interactions, generalizing to more complex environments like contextual bandits and Markov decision processes, and evaluating these LLM agents in real-world applications such as tool-use, web browsing, and software engineering.
This research marks an initial exploration into principled and novel post-training paradigms for LLMs in decision-making tasks. For more details, you can read the full research paper here.
