Advancing Autonomous AI Research Agents with Reinforcement Learning

TLDR: SFR-DeepResearch introduces a reinforcement learning framework to train autonomous single-agent large language models for complex Deep Research tasks. It features a flexible inference pipeline with minimal tools, a novel RL recipe using synthetic data and length normalization to stabilize training, and achieves state-of-the-art performance on benchmarks like Humanity’s Last Exam, outperforming many existing agents.

Artificial intelligence agents that can perform complex research tasks, often called Deep Research (DR), are a significant area of focus in AI. These agents need to search for information, reason over multiple sources, and use various tools to answer challenging questions or generate reports. A new research paper, “SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents,” introduces a novel approach to enhance the capabilities of single-agent systems for these demanding tasks.

The core idea behind this work is to improve autonomous single-agent models for Deep Research using continual reinforcement learning (RL). Unlike multi-agent systems, which rely on predefined roles and static workflows, a single agent dynamically determines its next action based on the current context, offering greater flexibility and potential for generalization to new tasks. The researchers from Salesforce AI Research have developed a simple yet effective RL recipe using entirely synthetic data, which they applied to various open-source large language models (LLMs).

Understanding Deep Research Agents

Deep Research agents are designed to interact with the real world through tools like web search and code interpreters. This interaction helps reduce hallucinations and improves reliability on complex, long-horizon tasks. The paper distinguishes between single-agent and multi-agent DR systems. Single-agent systems, like the ones developed in this research, involve a single, tool-equipped LLM that autonomously makes decisions without external directives at intermediate steps. Multi-agent systems, conversely, typically involve multiple agents with specialized roles collaborating within a predefined workflow.

The authors believe that single agents can generalize better to unseen tasks due to their flexible design. Furthermore, they can be seamlessly integrated as specialized sub-agents within more complex multi-agent frameworks, potentially simplifying overall system design.

The SFR-DeepResearch Framework

The SFR-DeepResearch framework focuses on continually training reasoning-optimized models to enhance their agentic skills while preserving their strong reasoning abilities. This framework consists of two main aspects:

• Agentic Inference Pipeline: This pipeline mimics a multi-turn conversation with tools and includes a sophisticated memory management system. Agents are equipped with a minimal set of tools: search_internet, browse_page, and code_interpreter. Notably, the browse_page tool strips hyperlinks, forcing the agent to discover new URLs solely through the search engine. For models like QwQ-32B and Qwen3, the multi-turn interaction is reframed as an iterative single-turn contextual question-answering problem, where previous tool calls and responses are packed into the user turn as memory. This approach helps these models, which are often optimized for single-step tasks, perform better in long multi-turn scenarios. For models like gpt-oss, which have stronger multi-turn abilities, the default chat template is used. The system also includes fault tolerance mechanisms to repair or retry faulty tool calls, guiding the model back on track.
• RL Training Recipe: The researchers developed a novel pipeline to synthesize complex, search- and reasoning-intensive training datasets. These datasets are designed to be more challenging than existing open-source options. The reinforcement learning algorithm used is based on REINFORCE, with modifications to stabilize policy optimization. A key innovation is the temporal advantage normalization, which prevents longer, potentially lower-quality trajectories from dominating the training loss. This ensures that the agent learns effective tool usage rather than simply making more calls. Strategic trajectory filtering and the reuse of partial rollouts further contribute to stable training.

Also Read:

• Reinforcement Learning: The Core Driver for Advanced AI Research Systems
• RLFactory: Empowering LLMs with Advanced Multi-Turn Tool Use

Performance and Insights

The SFR-DeepResearch agents were evaluated on three benchmarks: FRAMES, GAIA, and Humanity’s Last Exam (HLE). The best variant, SFR-DR-20B, trained from gpt-oss-20b, achieved up to 28.7% on the full text-only benchmark of Humanity’s Last Exam, demonstrating a significant improvement over its base model and competitive scores against other proprietary and open-source agents. The 8B and 32B variants also showed strong performance.

Key analyses revealed several insights:

• Modified Agentic Workflow: For models like QwQ-32B and Qwen3-8B, reframing multi-turn conversations into a single-turn contextual format led to significant performance gains, particularly a 10% absolute increase on FRAMES for the 32B model. This is because it aligns the task more closely with how these models were originally trained for single-step reasoning.
• Tool Usage and Normalization: The research found that simply increasing tool usage does not equate to better scores. Without the proposed length-normalized advantage modification, agents degenerated into repetitive, identical tool calls. The normalization term successfully regulated tool usage, leading to more stable training and improved performance.
• Response Length: SFR-DR-20B proved to be highly token-efficient, generating 4-5 times fewer tokens per step than the Qwen-family models. This indicates that gpt-oss-20b is better primed for agentic training and becomes even more efficient after RL training.

In conclusion, SFR-DeepResearch presents a compelling approach to developing autonomous single-agent systems for complex research tasks. By combining a tailored agentic workflow with an innovative reinforcement learning recipe, the researchers have successfully enhanced the agentic capabilities of reasoning-optimized LLMs. For more details, you can read the full research paper here.