TLDR: AGENTRL is a new framework that enables large language models (LLMs) to learn more effectively in complex, multi-turn, and multi-task environments. It features an asynchronous training pipeline and containerized environments for efficiency, along with novel algorithms like cross-policy sampling for better exploration and task advantage normalization for stable multi-task learning. Experiments show AGENTRL outperforms existing state-of-the-art LLMs across various agentic tasks and demonstrates strong generalization capabilities.
A new research paper introduces AGENTRL, a groundbreaking framework designed to significantly advance how large language models (LLMs) learn and operate as intelligent agents. This framework tackles the complex challenges of training LLM agents in environments where they need to interact multiple times (multi-turn) and handle a variety of different tasks simultaneously (multi-task).
Traditionally, training LLMs with reinforcement learning (RL) has been limited to simpler, single-turn interactions and often for one specific task at a time. However, real-world agentic tasks demand more: agents must engage in dynamic, extended interactions and adapt to diverse scenarios. AGENTRL steps up to meet these demands by providing a scalable infrastructure and stable training algorithms.
The Infrastructure Behind AGENTRL
One of AGENTRL’s core innovations is its infrastructure, built for efficiency and flexibility. It features a fully-asynchronous generation-training pipeline. This means that the process of generating agent actions and training the model happen independently and concurrently, drastically reducing idle time for powerful GPUs and speeding up multi-turn RL training. Imagine a factory where different stages of production run at their own pace without waiting for each other – that’s the efficiency AGENTRL brings.
To support a wide array of tasks, AGENTRL introduces a unified function-call based API interface. This standardizes how different environments communicate with the agent, making it easier to integrate diverse tasks. Each task environment is also containerized, meaning it runs in its own isolated unit. This improves resource management, prevents issues in one task from affecting others, and allows for seamless deployment across different hardware. A centralized controller oversees all these containerized environments, managing thousands of parallel training episodes.
Algorithmic Breakthroughs for Stability and Exploration
Beyond infrastructure, AGENTRL also brings novel algorithmic solutions. For multi-turn settings, where agents need to explore a vast number of possible actions, the framework proposes a “cross-policy sampling” strategy. This technique uses multiple LLMs to generate actions within a single interaction sequence, increasing the diversity of actions explored and preventing the model from getting stuck in repetitive or suboptimal behaviors. It’s like having several expert opinions guiding the agent’s choices, leading to more robust learning.
For multi-task training, AGENTRL introduces “task advantage normalization.” Agentic tasks can vary greatly in difficulty and the length of interactions required. This heterogeneity can cause training instability, with some tasks learning quickly while others lag. Task advantage normalization balances the learning process across different tasks by adjusting the reward signals, ensuring that all tasks contribute effectively to the agent’s overall improvement and preventing one task from dominating the training.
Also Read:
- AutoContext: Learning Environment Facts for Smarter AI Agents
- AutoMaAS: A Self-Evolving Framework for Multi-Agent AI Systems
Impressive Performance Across Diverse Tasks
The effectiveness of AGENTRL was demonstrated through extensive experiments. Trained on open LLMs (like Qwen2.5 and GLM-4-9B) across five agentic tasks—ALFWorld, DB, KG, OS, and Webshop—AGENTRL significantly outperformed leading models such as GPT-5, Claude-Sonnet-4, and DeepSeek-R1. The framework achieved a state-of-the-art average success rate of 70.4%.
Notably, a single model trained with AGENTRL across all five tasks achieved performance comparable to the best results from five separate models, each trained specifically for an individual task. This highlights AGENTRL’s ability to foster generalizable skills without sacrificing peak performance on specific tasks. Furthermore, the framework showed promising generalization capabilities, improving performance on unseen tasks like BFCL-v3.
The research paper also includes ablation studies, which confirmed the individual contributions of cross-policy sampling and task advantage normalization to the overall performance gains. These components were shown to be crucial for enhancing exploration and stabilizing multi-task training, respectively.
AGENTRL represents a significant leap forward in agentic reinforcement learning, providing a robust and scalable framework for training generalist LLM agents capable of handling complex, multi-turn, multi-task environments. The framework is open-sourced, inviting further development and application by the AI community. You can find more details about this work at the following link: AGENTRL Research Paper.
