TLDR: HAEPO (History-Aggregated Exploratory Policy Optimization) is a new reinforcement learning method designed to improve exploration and stability in policy optimization, especially for complex, long-horizon tasks. It works by evaluating and weighting entire decision sequences (trajectories) using a Plackett-Luce softmax, combined with entropy and KL regularization to prevent premature convergence and ensure stable learning. Empirical results show HAEPO converges faster, explores more thoroughly, and achieves more consistent and resource-efficient learning compared to existing methods like PPO, GRPO, and DPO across various benchmarks, including LLM fine-tuning.
In the rapidly evolving world of artificial intelligence, particularly in areas like reinforcement learning and large language models, effective exploration is crucial. Agents need to try new actions and strategies to discover optimal ways to achieve their goals. However, existing methods often struggle with balancing this exploration with stable learning, especially in tasks that involve many steps or complex decision-making over time.
A new research paper introduces a novel approach called History-Aggregated Exploratory Policy Optimization, or HAEPO. This method aims to overcome the limitations of current techniques like Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), which can sometimes restrict exploration in long-term tasks. HAEPO offers a history-aware exploratory loss function designed to encourage broader and more stable exploration.
The core idea behind HAEPO is to look at entire sequences of decisions, or ‘trajectories,’ rather than just individual steps or tokens. It compresses each trajectory into a single score based on the sum of its logarithmic probabilities. These scores are then used with a Plackett-Luce softmax function to assign normalized weights to each trajectory. This means that trajectories leading to higher rewards are given more credit, naturally encouraging the system to explore more diverse and successful paths.
To ensure stability and prevent the learning process from collapsing too quickly onto a few good trajectories, HAEPO incorporates two key regularization techniques. First, it adds an entropy regularization term, which acts like a ‘keep-searching’ pressure, flattening the distribution of trajectory weights and sustaining exploration. Second, it includes a soft KL penalty, which acts as a ‘speed governor’ on policy shifts, preventing the system from making abrupt changes that could lead to unstable learning. This penalty compares the current policy’s trajectory weights to those of a frozen reference policy, ensuring updates remain within a trusted region.
HAEPO offers several advantages. It maximizes exploration by amplifying diverse, high-return trajectories. It boasts high update efficiency, allowing for many iterations in a short amount of time, which translates to broader state-action coverage. Furthermore, it is resource-frugal, minimizing GPU memory and computational demands, making it scalable for long-horizon training.
Empirical evaluations of HAEPO across various environments demonstrate its effectiveness. In multi-armed bandit tasks, HAEPO showed increased exploration with more options and gained stability as the number of arms grew, all while converging quickly. For sparse, long-horizon tasks like a random walk, HAEPO achieved fast convergence, improved stability with increasing state space complexity, and was more computationally efficient than PPO and DPO. In the classic CartPole environment, HAEPO delivered competitive sample efficiency and, more importantly, produced significantly more stable and predictable policies, highlighting the critical role of its regularization terms.
Perhaps most notably, HAEPO was applied to fine-tuning large language models (LLMs) for the TL;DR summarization task. It achieved higher average human preference ratings and lower variance compared to GRPO and DPO, indicating more consistent alignment with human judgments. It also trained faster and consumed less GPU memory than GRPO, offering a favorable efficiency-performance trade-off. For more technical details, you can refer to the full research paper: HAEPO: History-Aggregated Exploratory Policy Optimization.
While HAEPO shows great promise, the authors acknowledge certain limitations. These include sensitivity to hyper-parameters, potential challenges with extremely long horizons (e.g., 10,000+ steps), computational overhead with very large batch sizes due to the listwise softmax, and its current focus on single-agent environments. Future work aims to address these areas, including exploring automated tuning for regularization scalars and extending the method to multi-agent settings.
Also Read:
- Proximal Supervised Fine-Tuning: Stabilizing LLM Updates for Broader Capabilities
- Stable LLM Training in Decentralized Networks: The HeteroRL and GEPO Approach
In conclusion, HAEPO provides a stable and interpretable framework for reinforcement learning by explicitly leveraging full-trajectory history while effectively balancing exploration and stability. Its ability to converge fast, explore thoroughly, and align closely with true rewards positions it as a significant advancement in policy optimization.
