TLDR: LEXPOL (Lexical Policy Networks) is a new multi-task reinforcement learning algorithm that uses natural language descriptions to guide an agent. It employs a text encoder and a learned gating module to select and blend multiple sub-policies, effectively combining fundamental skills to solve complex tasks. Evaluated on MetaWorld benchmarks, LEXPOL matches or exceeds existing methods in success rate and sample efficiency. A hybrid approach, LEXPOL + CARE, further improves performance by combining both skill and state factorization.
Multi-task reinforcement learning (MTRL) aims to train a single intelligent agent that can tackle many different tasks and effectively reuse skills across them. This field often uses task-specific information, like short descriptions in natural language, to help guide the agent’s behavior across various objectives. However, current methods don’t always fully capture how humans learn and combine skills.
A new research paper introduces Lexical Policy Networks, or LEXPOL, a novel approach to multi-task reinforcement learning. Developed by Rushiv Arora from the University of Massachusetts Amherst, LEXPOL is a language-conditioned architecture that uses a mixture of policies. The core idea is to encode task descriptions using a text encoder and then employ a learned ‘gating’ module to select or blend different sub-policies. This allows for end-to-end training across a wide range of tasks.
Understanding LEXPOL’s Approach
The motivation behind LEXPOL stems from how humans learn. We often master several smaller, fundamental skills and then combine them in various ways to solve new, more complex tasks. LEXPOL mirrors this by breaking down complex multi-task problems into these fundamental, reusable skills. Instead of a single, universal policy trying to handle everything, LEXPOL uses multiple sub-policies, each potentially specializing in a smaller skill.
The architecture of LEXPOL consists of three main components:
- Context Encoder: This component takes the natural language instruction (metadata) for a task and converts it into a fixed-dimension numerical representation. It uses pre-trained language models like BERT for this purpose.
- Mixture of Policies: This is a collection of ‘k’ different policies, each designed to learn and produce actions for smaller, factorized skills. All these policies receive the same state information from the environment.
- Gating Module: A multi-layer perceptron (MLP) takes the encoded language context and transforms it into ‘gating weights’. These weights act like a soft attention mechanism, determining how much each sub-policy’s output contributes to the final action taken by the agent.
This entire system can be trained from start to finish, allowing the agent to learn both the individual skills and how to combine them based on language instructions.
Comparison and Performance
LEXPOL draws comparisons to previous work like Context-Aware Representations (CARE), which also uses natural language but focuses on gating over state representations rather than policies. While CARE disentangles state information into object-specific representations, LEXPOL disentangles tasks into modular skills. The paper highlights that LEXPOL’s approach aligns more closely with the human tendency to combine discrete behaviors.
The researchers evaluated LEXPOL on the MetaWorld domain, a popular benchmark for robotics manipulation tasks, including MT10 (10 tasks) and MT50 (50 tasks). The results demonstrate that LEXPOL consistently matches or surpasses strong multi-task baselines in terms of success rate and how efficiently it learns (sample efficiency), even without needing task-specific retraining. For instance, on the MT10 setup after 2 million timesteps, LEXPOL achieved a success rate of 0.86, outperforming CARE’s 0.82 and other methods.
An interesting experiment involved a ‘frozen-experts’ setting, where sub-policies were pre-trained and fixed. LEXPOL was then trained only to learn the gating module. It successfully composed these pre-trained expert skills to solve new, composite tasks, like navigating to a red goal then a blue goal, demonstrating its ability to effectively combine existing knowledge using language cues.
Also Read:
- AGENTRL: A Scalable Framework for Training Generalist LLM Agents
- Teaching Robots Complex Skills: A New Approach to Grounded Skill Discovery
The Hybrid Approach: LEXPOL + CARE
The paper also proposes and tests a hybrid method called LEXPOL + CARE, which combines the strengths of both approaches. This method not only factorizes the state into its core components (like CARE) but also uses a selection of factorized modular skills (like LEXPOL). Experiments showed that LEXPOL + CARE achieved even higher success rates on MetaWorld benchmarks after extensive training (e.g., 0.90 on MT10 after 2 million timesteps), indicating that leveraging both state and policy disentanglement can lead to further improvements in multi-task reinforcement learning.
This research underscores the power of natural language metadata in guiding complex multi-task agents, offering a promising direction for creating more adaptable and human-like AI systems. You can read the full research paper here.
