TLDR: PLANNER-R1 is a new agentic RL method that uses reward shaping to significantly improve the performance of smaller (8B) language models on complex planning tasks like TRAVELPLANNER. It achieved a 56.9% pass rate, outperforming GPT-5, and was 3.5x more compute-efficient than 32B models. The gains generalize well to other tasks, showing that dense, process-level rewards are key for efficient and effective agentic RL with compact LLMs.
A new research paper introduces PLANNER-R1, an innovative approach to agentic reinforcement learning (RL) that significantly boosts the performance of smaller language models (LLMs) in complex planning tasks. This method, detailed in the paper by Siyu Zhu and colleagues from LinkedIn Corporation, demonstrates that strategic reward shaping can make 8-billion-parameter models competitive with much larger counterparts, while being considerably more efficient.
The core of PLANNER-R1’s success lies in its application of reward shaping, a technique where models receive more granular, process-level feedback during training, rather than just a sparse, final success signal. This dense feedback proved particularly effective for smaller LLMs (8B parameters), enabling them to achieve high final-pass rates on the challenging TRAVELPLANNER benchmark. For instance, PLANNER-R1 achieved a 56.9% final-pass rate, a remarkable 2.7 times improvement over GPT-5’s 21.2% baseline, making it the strongest agentic result on the public leaderboard.
One of the most compelling findings is the efficiency gain. The 8B models, when trained with dense reward shaping, were 3.5 times more compute-efficient and 1.5 times more memory-efficient than 32B models. This means that high performance in agentic RL doesn’t necessarily require massive models, opening doors for more accessible and sustainable AI development. While larger models (32B) were more robust under sparse rewards, they showed less relative improvement from shaping and higher variability across different training runs.
The TRAVELPLANNER benchmark itself is a rigorous test for LLM agents, requiring them to coordinate various tools like calendars, maps, and booking APIs to create multi-day travel itineraries. Agents must satisfy both explicit user constraints and implicit commonsense rules, a task where even powerful models like GPT-4-Turbo initially struggled, achieving only a 0.6% final pass rate. PLANNER-R1 redefines this challenge as a multi-step, tool-use Markov Decision Process, where the agent learns to gather facts, resolve conflicts, and output structured itineraries.
The reward function in PLANNER-R1 is multi-staged, incorporating indicators for schema compliance, satisfied commonsense constraints, and satisfied hard constraints. By adjusting the weighting of these “micro” and “macro” rewards, the researchers could control the density of feedback. This proper reward shaping ensures that intermediate guidance is provided, while the ultimate objective of task success is still met. The study also explored curriculum learning, but found that reward shaping alone consistently amplified learning dynamics more effectively.
Crucially, the performance gains from PLANNER-R1 did not come at the cost of generalization. The fine-tuned models maintained or even surpassed their baseline performance on out-of-domain tasks such as MULTI-IF, NATURALPLAN, and Ï„-BENCH. This suggests that the efficiency achieved through agentic RL with reward shaping is robust and transferable to diverse planning and tool-use scenarios, without leading to overfitting to the training data.
Also Read:
- RoRecomp: Making LLMs Reason More Concisely and Efficiently
- Unlocking New Abilities: How Reinforcement Learning Helps Language Models Compose Skills
The research highlights the importance of reward design as a critical factor for scaling agentic RL and underscores the competitive strength of smaller models when guided effectively. This work was conducted by researchers at LinkedIn Corporation. You can read the full paper here.
