TLDR: EVOR (Expressive Value Learning for Offline Reinforcement Learning) is a new approach that makes offline reinforcement learning more scalable by integrating expressive policies and expressive value functions. It avoids computationally expensive backpropagation through time and error-prone policy distillation. EVOR learns a base policy via flow matching and an optimal, regularized Q-function through flow-based temporal difference learning. During inference, it uses rejection sampling against this expressive value function, allowing for efficient policy extraction, regularization, and scaling without retraining. Experimental results show EVOR outperforms baselines on diverse robotics tasks, demonstrating the benefits of its expressive value learning approach.
Reinforcement Learning (RL) has shown immense potential in various fields, from fine-tuning large language models to complex decision-making tasks. However, its application in robotics has faced significant hurdles, primarily due to scalability issues. Traditional online RL methods require extensive real-world interactions, which are often costly, time-consuming, and can pose safety risks. This challenge has led to the rise of offline RL, where agents learn from pre-collected, diverse datasets without further environmental interaction.
Addressing Scalability in Offline RL
Scaling offline RL to handle increasingly complex datasets demands the use of expressive generative models, such as diffusion and flow matching. These models can capture intricate data distributions, which is crucial when dealing with datasets generated by multiple policies of varying quality. However, existing methods that leverage these powerful models often rely on either backpropagation through time (BPTT) or policy distillation. BPTT is computationally intensive and memory-demanding, potentially degrading the general knowledge of the underlying base policy. Policy distillation, while offering a one-step model for efficiency, introduces compounding errors and limits scalability to larger, more complex base policies like vision-language-action (VLA) models.
The core question then becomes: how can we develop a scalable offline RL approach that avoids these limitations?
Introducing EVOR: Expressive Value Learning for Offline Reinforcement Learning
A new research paper, “Expressive Value Learning for Scalable Offline Reinforcement Learning”, introduces EVOR, a novel approach designed to overcome these challenges. Authored by Nicolas Espinosa-Dice and Wen Sun from Cornell University, and Kianté Brantley from the Kempner Institute at Harvard University, EVOR integrates both expressive policies and expressive value functions to achieve scalable offline RL without relying on distillation or BPTT.
How EVOR Works
EVOR’s methodology is built on two key pillars:
First, it learns an expressive base policy through flow matching. In scenarios where a pre-trained base policy isn’t available, EVOR trains one by predicting actions via behavioral cloning on the offline dataset’s state-action pairs. Flow matching, a powerful generative model, allows this base policy to model complex and multi-modal action distributions present in the offline data. This effectively approximates the unknown data-generating policy.
Second, and most distinctively, EVOR learns an expressive, optimal Q-function using a technique called flow-based Temporal Difference (TD) learning. Instead of traditional value learning methods, EVOR integrates flow matching into distributional TD learning. This allows it to leverage the expressivity of flow matching while benefiting from the variance reduction and improved credit assignment of TD learning. Essentially, flow matching is used to transport samples from a prior distribution to a target distribution, which in EVOR’s case, is the distribution of rewards-to-go under the reference policy. This results in learning an optimal, regularized Q-function.
During inference, EVOR employs an innovative inference-time policy extraction method. Instead of learning a new policy during training, it optimizes the base policy by performing rejection sampling against the learned expressive value function. Unlike standard rejection sampling methods that use unregularized value functions, EVOR utilizes its provably optimal and regularized Q-function. This enables efficient optimization, regularization, and compute-scalable search without requiring any retraining.
Key Advantages and Scalability
EVOR offers several significant advantages for scalable offline RL:
- Avoids Distillation and BPTT: By using inference-time policy extraction with an optimal, regularized Q-function, EVOR bypasses the computational and error-prone issues of backpropagation through time and policy distillation.
- Expressive Value Functions: It learns an expressive, optimal Q-function via flow matching, bringing the same level of expressivity to value functions that generative models have brought to policies.
- Inference-Time Scaling and Regularization: EVOR provides a natural mechanism for scaling by allowing more action candidates to be evaluated during inference, guided by the expressive value function. Crucially, it can also adjust the level of regularization by varying temperature parameters at test-time, all without additional training. This flexibility allows for adapting to different compute budgets and performance requirements.
- Consistent Performance Across Tasks: A notable benefit is that EVOR uses the same training and evaluation parameters across a diverse set of environments and tasks, simplifying its application compared to many policy gradient-based algorithms that require environment-specific tuning.
Also Read:
- Understanding Goals Through Temporal Distances: A New Approach to Reinforcement Learning
- Teaching Robots Complex Skills: A New Approach to Grounded Skill Discovery
Experimental Validation
The researchers evaluated EVOR on a diverse set of offline RL tasks from the OGBench task suite, including locomotion and manipulation robotics. Empirically, EVOR consistently outperformed baselines, including Q-chunking (QC), a closely related algorithm that also uses flow matching for the base policy and rejection sampling for policy extraction, but with a standard value function. The results demonstrate the clear benefit of integrating expressive value learning into offline RL, showing improved performance across 25 unique tasks.
In essence, EVOR represents a significant step forward in making offline reinforcement learning more scalable and practical for complex real-world applications, particularly in robotics, by intelligently leveraging expressive generative models for both policies and value functions.
