TLDR: This paper introduces an iterative framework for automated reward design in reinforcement learning, specifically for the Gran Turismo 7 racing game. It leverages large language models (LLMs) to generate reward functions and video language models (VLMs) for preference-based evaluation, reducing the need for manual reward engineering. The system produces racing agents competitive with or surpassing GT Sophy and can generate novel driving behaviors, demonstrating a practical approach to automated reward design in complex environments.
Designing artificial intelligence (AI) agents that can perform complex tasks, like racing in a video game, often hinges on how well we can define what ‘success’ looks like. In the world of reinforcement learning (RL), this definition comes in the form of a ‘reward function’ – a numerical feedback system that tells the AI when it’s doing well or poorly. However, crafting these reward functions, especially for intricate environments like autonomous racing, is notoriously difficult and time-consuming, even for experts.
A recent research paper, Automated Reward Design for Gran Turismo, tackles this challenge head-on. Authored by Michel Ma, Takuma Seno, Kaushik Subramanian, Peter R. Wurman, Peter Stone, and Craig Sherstan, the paper introduces an innovative system that uses modern foundation models to automate the creation of these crucial reward functions for the Gran Turismo 7 racing game.
The Challenge of Reward Design
Imagine trying to teach a child to ride a bike. You wouldn’t just tell them to ‘ride well’; you’d give them specific feedback: ‘push the pedals,’ ‘steer straight,’ ‘don’t fall.’ In RL, reward functions are that feedback. But translating human-understandable goals like ‘win races while maintaining good sportsmanship’ into precise numerical rewards that an AI can optimize is a significant hurdle. Traditional methods often involve extensive trial and error, requiring deep expertise in both RL and the specific domain.
Leveraging Foundation Models for Automation
The researchers propose an iterative framework that combines the power of large language models (LLMs) and video language models (VLMs) to streamline this process. The system works by taking text-based instructions from a user and then automatically generating, evaluating, and refining reward functions.
- LLM-based Reward Generation: At the core, an LLM is prompted with the environment details (like the car’s speed and position) and the desired task (e.g., ‘win races while obeying standard motorsport rules’). The LLM then generates Python code for various reward components, such as rewards for making progress or penalties for collisions.
- VLM Preference-based Evaluation: Instead of relying on a pre-defined ‘fitness metric’ (which can be as hard to design as the reward function itself) or constant human oversight, the system uses a VLM. After an AI agent is trained with a generated reward function, the VLM watches videos of the agent’s performance and provides pairwise preferences, essentially judging which agent is performing better according to the given task description. This acts as an automated replacement for human expertise.
- Trajectory Alignment Filter: To save computational resources, a ‘trajectory alignment filter’ is used. This filter, based on a statistical correlation, helps prune out reward functions that are unlikely to lead to good performance before expensive training even begins, by checking their alignment with previously collected VLM preferences.
- Human Feedback (Optional but Beneficial): While largely automated, the system can still incorporate human feedback in text form, which the LLM uses to refine its reward generation in subsequent iterations.
Outperforming the Best
The experiments were conducted in Gran Turismo 7, using a Mercedes-AMG GT3 ’16 on the Lake Maggiore track. The benchmark for performance was GT Sophy, a champion-level RL racing agent developed by Sony AI, known for outracing top human competitors. The results were impressive: the automated reward design system produced racing agents that were not only competitive with GT Sophy but sometimes even surpassed its performance.
The framework, referred to as ‘Ours’ in the paper, consistently outperformed simpler approaches like ‘Text2Reward’ (which generates only one reward per iteration) and a ‘No feedback’ system (which relies solely on LLM-generated feedback without human input). This highlights the importance of the iterative refinement and the VLM’s role in accurate evaluation.
VLM as a Human Expert Replacement
A crucial finding was the accuracy of the VLM in evaluating agent performance. The VLM’s preferences aligned closely with human expert judgments, demonstrating its capability to effectively replace manual fitness metric design. Interestingly, the VLM performed best when given both trajectory data and video footage, indicating a current gap in vision-only models compared to human perception.
Generating Novel Behaviors
Beyond achieving superhuman performance in standard racing, the system also demonstrated its ability to generate novel behaviors. The researchers successfully prompted the AI to ‘race as fast as possible in reverse at all times’ and ‘drift as much as possible while otherwise obeying standard motorsport rules,’ showcasing the flexibility and creative potential of automated reward design.
Also Read:
- Unlocking Advanced AI Reasoning for Creative Challenges
- Predicting Transit Delays: How AI Learns from Unstructured Alerts
Future Outlook
This research marks a significant step towards making reinforcement learning more accessible by automating one of its most challenging aspects: reward design. While the system is computationally intensive and still benefits from human input for consistent results, its components are not specific to Gran Turismo 7, suggesting broad applicability to other complex environments. The work paves the way for a future where AI agents can learn desired behaviors directly from natural language instructions, without the need for extensive manual engineering of their reward systems.
