TLDR: This research paper investigates why discrete Soft Actor-Critic (DSAC) performs poorly in discrete-action off-policy reinforcement learning compared to value-based methods like DQN. The authors identify that the coupling of actor and critic entropy regularization is the primary reason. By decoupling these components and introducing a flexible new actor-critic framework, they demonstrate that their proposed methods can achieve performance comparable to DQN on Atari games, even without explicit entropy regularization or exploration, provided the actor objective is sufficiently optimized.
Reinforcement learning, particularly in environments with discrete actions like classic Atari games, has long been dominated by value-based algorithms such as Deep Q-Networks (DQN). These methods are highly effective at learning from previously collected data, making them very efficient. However, policy-based methods, which directly learn a strategy, have often struggled in this off-policy, discrete-action setting. On-policy methods like PPO don’t reuse data efficiently, while off-policy ones like Soft Actor-Critic (SAC) have shown poor performance in their discrete variant (DSAC).
A recent research paper, “Revisiting Actor-Critic Methods in Discrete Action Off-Policy Reinforcement Learning” by Reza Asad, Reza Babanezhad, and Sharan Vaswani, delves into this disparity. The authors set out to understand why DSAC underperforms and to design simpler, more effective actor-critic methods that can match DQN’s performance in discrete-action environments.
Uncovering the Core Issue in DSAC
Previous attempts to improve DSAC focused on issues like fixed target entropy or Q-value underestimation. However, this new study reveals a more fundamental problem: the tight coupling between the actor and critic entropy regularization. Entropy regularization encourages exploration by making policies more random. In DSAC, when both the actor (the component that decides actions) and the critic (the component that evaluates actions) use entropy regularization in a coupled manner, it leads to instability and poor results.
The researchers found that by simply disabling the entropy regularization in the critic update – essentially using a “hard Bellman operator” for policy evaluation – DSAC’s performance dramatically improved. This modified DSAC variant, without needing complex additions like double Q-learning or adaptive entropy targets, achieved results comparable to DQN across various Atari games. This key insight highlighted that the critic’s entropy coefficient significantly impacts DSAC’s empirical performance.
A Flexible New Framework for Actor-Critic Methods
Motivated by this discovery, the paper introduces a general off-policy actor-critic framework that offers greater flexibility. This framework decouples the actor’s entropy regularization from the critic’s, allowing them to be controlled independently. It also incorporates new ways to update the actor’s policy.
The framework’s policy evaluation step for the critic can use an m-step Bellman operator, which can be either “soft” (with entropy regularization, controlled by a ‘critic entropy’ coefficient ζ) or “hard” (without entropy regularization). For the actor’s policy optimization, it involves a two-stage process: first, computing an intermediate policy using methods like Natural Policy Gradient (NPG) or Softmax Policy Mirror Ascent (SPMA), and then projecting this policy onto the set of realizable policies using either forward or reverse KL divergence, while also applying a “proximal entropy regularization term” (actor entropy, Ï„).
Theoretical Backing and Empirical Validation
The authors provide theoretical guarantees for their proposed methods in a simplified tabular setting, demonstrating that these methods can converge to the optimal regularized value function. This theoretical analysis supports the design choices and the decoupling of actor and critic entropy.
Empirically, the framework’s objectives were tested on Atari 2600 games, yielding several important findings:
- Consistent with the DSAC ablation, using the hard Bellman operator (ζ = 0) for the critic significantly improved performance across all proposed objectives.
- Surprisingly, these new objectives could achieve performance competitive with DQN even without any entropy regularization (τ = 0 and ζ = 0) or explicit exploration strategies. This was possible when the actor objective was sufficiently optimized, meaning more gradient steps were taken during its update. This suggests that sufficient optimization can compensate for the lack of explicit exploration.
- The choice between forward and reverse KL divergence for projecting the intermediate policy generally did not have a significant impact on performance when actor entropy was present. However, in the absence of entropy regularization, forward KL showed some advantages for certain actor updates.
Also Read:
- AI Algorithm Optimizes Complex Scheduling from Imperfect Historical Data
- Information Theory Unlocks Deeper Understanding and Diagnosis of Reinforcement Learning Agents
Looking Ahead
This research provides a fresh perspective on designing off-policy actor-critic methods for discrete action spaces. By identifying and addressing the issue of coupled entropy regularization, the authors have paved the way for simpler, yet highly effective, algorithms that can rival the performance of established value-based methods like DQN. The flexibility of their new framework also opens doors for future work, including extending it to continuous-action environments and further exploring the behavior of these objectives without explicit entropy regularization.
