TLDR: PolicyEvolve is a new framework that uses Large Language Models (LLMs) to create and continuously improve understandable, rule-based policies for complex multi-player games. Unlike traditional methods that require vast data and lack transparency, PolicyEvolve employs a dual-pool architecture (Global and Local Pools) and an iterative refinement process. This allows it to autonomously evolve high-performance strategies with minimal environmental interaction, leading to more robust and interpretable AI agents, as demonstrated in experiments on a sumo robot game.
Multi-agent reinforcement learning (MARL) has shown great promise in solving complex multi-player games, often through agents learning by playing against themselves. However, developing effective strategies in MARL typically demands vast amounts of training data and significant computing power. A major drawback of these advanced strategies is their lack of transparency, making it hard for humans to understand how decisions are made, which can hinder their use in real-world situations.
Recently, a new approach has emerged where Large Language Models (LLMs) are used to create programmatic policies for single-agent tasks. These policies are essentially sets of rules or code, which are much easier to understand than the complex internal workings of neural networks. This shift from ‘black-box’ neural networks to ‘white-box’ rule-based code also brings efficiency benefits.
Inspired by these developments, researchers have introduced PolicyEvolve, a new framework designed to generate programmatic policies specifically for multi-player games. PolicyEvolve aims to significantly reduce the need for manually written policy code and achieve high-performing strategies with minimal interaction with the game environment.
How PolicyEvolve Works
The PolicyEvolve framework is built around four main components:
- Global Pool: This acts as a repository for the best-performing policies discovered throughout the training process. Think of it as a hall of fame for successful strategies.
- Local Pool: This temporarily stores policies that are currently being developed and refined in the ongoing training iteration. Only policies that prove to be sufficiently strong are promoted to the Global Pool.
- Policy Planner: This is the core engine for generating new policies. It takes inspiration from the top policies in the Global Pool, considers information about the game environment, and then refines its initial policy ideas based on feedback from the Trajectory Critic.
- Trajectory Critic: This module observes how a policy performs in the game, identifies its weaknesses or ‘vulnerabilities’, and then suggests specific improvements to guide the Policy Planner in creating better policies.
The process is iterative: the Policy Planner generates a new policy, which is then tested. The Trajectory Critic analyzes its performance, and the Policy Planner uses this analysis to make improvements. This cycle continues until the policy achieves a high enough win rate against the strategies in the Global Pool, at which point it is integrated into the Global Pool itself. This continuous evolution allows policies to adapt to dynamic multi-agent environments through self-play.
Key Advantages and Contributions
PolicyEvolve offers several significant contributions:
- It is the first programmatic reinforcement learning framework specifically designed for multi-agent tasks, capable of autonomously evolving policies to improve their quality consistently.
- It enhances policy robustness through its unique Global and Local policy pools, which are trained using a Population-Based Training approach.
- Experiments show that PolicyEvolve achieves superior efficiency in terms of samples needed and produces higher quality policies compared to other prompt-based methods.
Also Read:
- Training Language Models Without New Data: The Language Self-Play Approach
- Navigating Complexity: How New AI Framework Guides LLMs to Smarter Reasoning
Experiments and Results
The framework was tested extensively using various LLMs on a multi-player game called ‘Wrestle’, provided by the Chinese Academy of Sciences’ JIDI platform. In ‘Wrestle’, two sumo robot-like agents compete in a circular arena, trying to push each other out while managing their energy. The agents’ actions involve applying force and steering angle.
The results demonstrated that policies generated by PolicyEvolve consistently improved over 20 iterations, showing higher ELO scores (a rating system for skill levels) with each newer policy. When compared against state-of-the-art prompt-based techniques like Naive, CoT, and React, PolicyEvolve’s policies significantly outperformed them in ELO score and win rate. This indicates that PolicyEvolve can generate complex and effective programmatic policies.
Ablation studies also confirmed the importance of various design choices within PolicyEvolve. For instance, providing auxiliary information (like ‘boundary colors’) in the prompt significantly improved initial policy quality. A two-step approach for policy iteration (summarizing experience data first, then generating improvements) was found to be more effective than direct generation. Furthermore, storing historical reflections and improvement suggestions in a ‘Reflection Memory’ proved crucial for achieving higher win rates.
For more technical details, you can read the full research paper here.
