TLDR: This paper analyzes temporal-difference (TD) learning methods that learn both state and action values, categorizing them into QV-learning and AV-learning. It proves the convergence of on-policy QV-learning, identifies and corrects a bias in the off-policy QVMAX algorithm, and introduces Regularized Dueling Q-learning (RDQ). RDQ, a new AV-learning method, significantly outperforms Dueling DQN in deep reinforcement learning benchmarks, highlighting the benefits of learning two value functions.
In the dynamic field of reinforcement learning (RL), agents learn to make decisions by interacting with their environment to maximize cumulative rewards. A cornerstone of RL algorithms is Temporal-Difference (TD) learning, which uses predictions of future values to refine current value estimates—a process known as bootstrapping.
Traditionally, most TD methods for control, such as Q-learning and Sarsa, rely on a single action-value function, often denoted as Q(s, a), which estimates the expected return for taking a specific action in a given state. However, a less explored but promising paradigm involves learning two asymmetric value functions: one for state values (V(s)) and another for action values (Q(s, a)), where state values serve as an intermediate step in learning action values.
This research delves into two main families of these dual-value function approaches: QV-learning and AV-learning. The paper meticulously analyzes their theoretical soundness, convergence properties, and sample efficiency, shedding light on when and why learning two value functions might be more advantageous than learning just one.
Understanding QV-learning
QV-learning methods directly estimate both Q(s, a) and V(s), with a degree of bootstrapping between them. The classic QV-learning algorithm, for instance, enhances Expected Sarsa by incorporating a learned approximation of state values. The study reveals that for on-policy prediction (where the agent follows a fixed behavior policy), QV-learning can be significantly more efficient than Expected Sarsa, especially as the number of possible actions in an environment increases. This is because QV-learning can learn state values with fewer samples, making it more scalable to large action spaces. While this approach introduces a noisier update, it can be beneficial when a satisfactory value function is needed quickly with limited samples.
A significant contribution of the paper is the first published proof of the expected contraction of QV-learning for on-policy prediction. This theoretical finding confirms that both Q and V functions in QV-learning make progress towards their correct values, even when updates are asynchronous, addressing previous assumptions about their convergence.
However, the paper also addresses challenges in off-policy control, where the agent learns about an optimal policy while executing a different behavior policy. QVMAX, a prominent off-policy variant of QV-learning, was found to suffer from a critical bias. The research demonstrates that QVMAX generally does not converge to the optimal action and state values. To rectify this, the authors introduce Bias-Corrected QVMAX (BC-QVMAX), an unbiased algorithm that empirically restores convergence. Despite its theoretical correctness, BC-QVMAX was observed to be somewhat slower than Q-learning in experiments, suggesting that the smoothing effect of its updates might introduce a delay in bootstrapping.
Exploring AV-learning and the New RDQ Algorithm
AV-learning methods take a different approach by decomposing the action-value function into a state-value function and an advantage function (Adv(s, a)), where Q(s, a) = V(s) + Adv(s, a). This strategy was popularized in deep reinforcement learning by the Dueling Network Architecture. Unlike QV-learning, the two value functions in AV-learning do not directly bootstrap from each other but from the composite action-value function.
The paper formalizes a tabular version of Dueling Q-learning, providing deeper insights into its underlying mechanics. Building on this, the authors introduce a novel AV-learning algorithm called Regularized Dueling Q-learning (RDQ). A key innovation of RDQ is that it addresses the identifiability issue often present in dueling decompositions without needing to subtract a mean advantage term. Instead, RDQ uses an L2 penalty (in its Soft RDQ variant) or directly follows a minimum-norm path (in its Hard RDQ tabular variant) to find a unique and efficient solution for the value functions.
Empirical results from off-policy control experiments show that both Dueling Q-learning and Hard RDQ significantly outperform traditional Q-learning, highlighting the general effectiveness of the advantage decomposition. In deep RL experiments using the MinAtar benchmark (a suite of Atari-like games), RDQ demonstrated a remarkable performance advantage. Despite sharing an identical network architecture and well-tuned hyperparameters with Dueling DQN, RDQ consistently and significantly outperformed both DQN and Dueling DQN across all five MinAtar environments. This suggests that RDQ’s approach to regularization and value function decomposition leads to more effective and sample-efficient learning in complex deep RL settings.
Also Read:
- Advancing Policy Optimization: A New Algorithm for Efficient Reinforcement Learning
- A New Approach for Stable Learning in Diverse Multi-Agent AI Systems
Conclusion
This research significantly advances our understanding of TD-learning algorithms that leverage both state and action value functions. It clarifies the benefits of on-policy QV-learning for prediction, identifies and corrects a critical bias in QVMAX, and introduces RDQ, a powerful new Dueling Q-learning method. The findings underscore the potential of learning state values in tandem with action values to accelerate and improve the effectiveness of reinforcement learning algorithms. For more details, you can refer to the full research paper available at arXiv:2507.09523.
