TLDR: CodeGym is a novel framework that converts coding problems into diverse, verifiable, and interactive reinforcement learning environments for large language model (LLM) agents. By training LLM agents in these synthetic multi-turn tool-use environments, CodeGym significantly improves their ability to generalize to new tools and unseen workflows, outperforming traditional supervised fine-tuning methods and demonstrating consistent gains across various benchmarks.
Large language models (LLMs) have shown incredible abilities in reasoning and following instructions, but they often struggle when it comes to using external tools in real-world scenarios. These ‘LLM agents’ need to interact with databases, search engines, and code executors to truly solve diverse tasks. However, current training methods, which often rely on static examples or narrow reinforcement learning tasks, lead to agents that don’t generalize well to new tools or unfamiliar workflows.
A new research paper, titled “Generalizable End-to-End Tool-Use RL with Synthetic CodeGym,” introduces a groundbreaking solution: CodeGym. This scalable framework transforms static coding problems into dynamic, interactive environments specifically designed for training LLM agents using reinforcement learning (RL). The paper, authored by Weihua Du, Hailei Gong, Zhan Ling, Kang Liu, Lingfeng Shen, Xuesong Yao, Yufei Xu, Dingyuan Shi, Yiming Yang, and Jiecao Chen, highlights how CodeGym enables agents to actively explore and master various workflows, leading to significantly improved generalization.
The Challenge of Generalization for LLM Agents
Traditional training for LLM agents often involves supervised fine-tuning (SFT) on pre-recorded interactions. While this can improve performance on specific benchmarks, the agents tend to overfit these patterns and fail when faced with new tools or slightly different task configurations. Reinforcement learning offers a promising alternative, allowing agents to learn through active exploration and feedback, adapting to novel situations rather than just memorizing paths. However, a scalable and general-purpose RL environment for LLM agents has been largely absent.
Introducing CodeGym: A Synthetic Training Ground
CodeGym addresses this gap by leveraging the inherent logic and structure of coding problems. The core insight is that code execution mirrors many real-world workflows. By converting these problems into interactive environments, CodeGym provides a rich and diverse training ground. Instead of directly generating full code, LLM agents in CodeGym learn to invoke atomic functions or logic extracted from coding solutions as callable tools.
Imagine a coding problem like “finding the number closest to K in a sorted list.” CodeGym transforms this into an environment where an agent might have tools like `observe()` (to get list length and target K), `look_up_pos(i)` (to check an element at index i), and `done(answer)` (to submit the final answer). The agent then interacts with these tools, making calls and receiving feedback, until it solves the task. The environment provides a binary reward: 1 for a correct answer, 0 otherwise, making it ideal for RL.
How CodeGym Works: Synthesis and Verification
The CodeGym generation pipeline involves two main stages: Gym Synthesis and Gym Verification.
During **Gym Synthesis**, an LLM is prompted to extract reusable code logic from programming solutions and rewrite them into a library of modular tools, complete with precise documentation. These tools form the action space for the LLM agent. Each CodeGym environment is designed as a Partially Observable Markov Decision Process (POMDP), where the agent interacts through tool calls and receives sparse rewards upon task completion.
The **Gym Verification** stage is crucial for ensuring the quality and solvability of the synthesized environments. It generates a collection of unit tests covering various difficulty levels and edge cases. An LLM then generates candidate “solution functions” that use the available tools to solve the environment. If any of these candidate solutions pass all unit tests, the environment is deemed solvable and valid for training.
Ensuring High-Quality Training Data
CodeGym employs rigorous quality control filters to select high-quality training instances. These filters consider:
- **Tool-Use Complexity:** Environments must require a non-trivial number of tool calls (between 10 and 256) and involve at least 4 distinct tools to avoid overly simple or repetitive tasks.
- **Difficulty:** Training instances are filtered to ensure they are not too easy for agents, retaining only those where a baseline model (Qwen2.5-32B-Instruct) achieves an accuracy no greater than 25%.
For more advanced models, like long-Chain-of-Thought (CoT) models, CodeGym also includes a **Difficulty Augmentation** step. This involves scaling parameter values in test cases to large magnitudes and requiring more intricate environment logic, preventing models from bypassing tool calls by relying solely on internal reasoning.
A Robust Training Framework
CodeGym utilizes a distributed RL framework for high-throughput training. A central server manages environments, and rollout workers interact with these environments in real-time. A “trial-then-overwrite” mechanism ensures robustness: if an agent’s function call causes an environment crash, the environment state is rolled back, and an error message is returned as feedback, preventing training disruptions.
Impressive Generalization Results
Experiments with various language models, including the Qwen2.5 series and QwQ-32B, demonstrate CodeGym’s effectiveness. Models trained on CodeGym consistently show significant improvements in out-of-distribution (OOD) generalization across diverse benchmarks, including tool-use scenarios (Ï„-bench, Ï„2-bench), multi-turn interactions (ALFWorld), and even reasoning tasks (ZebraLogic, MMLU-Pro).
For instance, Qwen2.5-32B-Instruct achieved an absolute accuracy gain of 8.7 points on the OOD benchmark Ï„-Bench. Larger models generally benefited more from CodeGym training, indicating that increased model capacity can lead to stronger generalizability when combined with this training approach. The research also found that RL-based training on CodeGym significantly outperforms supervised fine-tuning methods in achieving OOD generalization, underscoring the value of active learning.
The average number of tool calls made by agents during training also increased, suggesting that agents learned to execute longer, more structured procedures, aligning closer with optimal solutions. This indicates an improved understanding of multi-step agent workflows.
Also Read:
- Reasoning Core: A Scalable Platform for Training LLMs in Foundational Logic
- Foundation Models Navigate Virtual Worlds: New Strategies for Reinforcement Learning
A Step Towards More Capable LLM Agents
CodeGym represents a significant advancement in training LLM agents. By providing a scalable, diverse, and verifiable environment for reinforcement learning, it helps agents develop robust and generalizable tool-use capabilities. This work lays a strong foundation for building more capable LLM agents that can effectively navigate and interact with the complexities of real-world tool-augmented workflows. You can find the full research paper at https://arxiv.org/pdf/2509.17325.
