TLDR: DreamGym is a new framework that uses a reasoning-based experience model to synthesize diverse and scalable interaction data for training large language model (LLM) agents with reinforcement learning (RL). It overcomes challenges like costly real-world rollouts and limited task diversity by generating consistent state transitions and adaptive tasks, significantly improving RL performance in both synthetic and sim-to-real scenarios while reducing real-world data requirements.
Training advanced AI agents, especially those powered by large language models (LLMs), to learn and improve through interaction has been a significant challenge. Traditional methods often face hurdles like expensive real-world interactions, a limited variety of tasks, inconsistent feedback, and complex infrastructure. These issues make it difficult to gather enough diverse data for effective reinforcement learning (RL).
A new framework called DreamGym has been introduced to tackle these problems. It’s designed to create varied experiences for AI agents in a scalable way, making online reinforcement learning training much more efficient. Instead of relying on costly interactions in real environments, DreamGym uses a “reasoning-based experience model” that simplifies how environments work into a step-by-step thinking process. This model generates consistent state changes and feedback signals, allowing for the collection of a vast amount of agent interaction data for RL training at a much lower cost.
DreamGym enhances the quality and stability of these interactions by using an “experience replay buffer.” This buffer starts with real-world data and is continuously updated with new synthetic interactions, actively supporting the agent’s learning. To ensure agents keep learning new things, DreamGym also intelligently creates new tasks that are specifically designed to challenge the agent’s current abilities. This adaptive task generation helps in building a more effective learning path for the agent.
The framework has been tested across various environments and with different types of LLM agents. The results show that DreamGym significantly improves RL training. For tasks not typically set up for RL, like WebArena, DreamGym agents performed over 30% better than other methods. In environments where RL is possible but very expensive, DreamGym achieved performance comparable to leading RL algorithms like GRPO and PPO, but solely using synthetic interactions.
Furthermore, DreamGym offers a “sim-to-real” transfer strategy. Agents first train extensively in the synthetic DreamGym environment and then transfer their learned skills to real-world settings. This approach led to substantial performance gains (over 40%) in real environments while requiring significantly fewer real-world interactions (less than 10% of the data needed for training from scratch). This makes DreamGym a promising way to “warm-start” general-purpose RL training efficiently.
The core of DreamGym’s success lies in its three main components: the reasoning experience model, the experience replay buffer, and the curriculum task generator. The reasoning experience model abstracts environment dynamics into a textual space, making it efficient to generate informative interactions. The replay buffer combines initial real-world knowledge with ongoing synthetic data, adapting as the agent learns. The curriculum task generator identifies and creates increasingly challenging tasks based on how well the agent is currently performing, ensuring continuous learning and exploration.
Also Read:
- RLoop: A Self-Improving Approach to Overcome Overfitting in Reinforcement Learning for LLMs
- The Three Pillars of Effective AI Agents: Productivity, Proactivity, and Personalization
This innovative approach addresses the long-standing challenges in training LLM agents with RL: the high cost of data collection, the lack of diverse tasks, unstable reward signals, and complex infrastructure. By creating an environment specifically designed for RL, DreamGym enables efficient synthetic training and effective transfer to real-world scenarios, improving generalization while minimizing reliance on costly real-world interactions. You can read the full research paper here: Scaling Agent Learning via Experience Synthesis.
