TLDR: Learning Progress Monitoring (LPM) is a novel intrinsic reward method for reinforcement learning that enables noise-robust exploration. Inspired by human learning, LPM rewards agents for improvements in their environment model rather than prediction error or novelty, effectively guiding them towards learnable transitions and away from unlearnable noise (the “noisy-TV problem”). Through a dual-network design, LPM theoretically aligns with information gain and empirically outperforms state-of-the-art baselines in noisy MNIST, 3D mazes, and Atari games, demonstrating faster convergence, broader exploration, and higher extrinsic rewards in stochastic environments.
Reinforcement learning agents often struggle with exploration in environments where rewards are few and far between. Imagine a robot trying to find its way out of a complex maze; it might only get a reward signal once it reaches the exit. To overcome this “sparse reward problem,” researchers have developed intrinsic rewards – internal motivators that encourage agents to explore systematically, even without external feedback.
However, a significant hurdle for these intrinsic reward methods is the “noisy-TV problem.” This occurs when an environment contains unlearnable sources of randomness, like static on a TV screen. A curious agent, driven by prediction error or novelty, might get stuck repeatedly observing this unpredictable noise, wasting valuable time and computational resources without actually learning anything useful about its environment. While some existing methods try to distinguish between learnable (epistemic) and unlearnable (aleatoric) uncertainty, this often requires vast amounts of data or strong prior knowledge, making early exploration inefficient and susceptible to noise.
Drawing inspiration from recent neuroscience findings that show humans naturally monitor their learning progress during exploration, researchers Zhibo Hou, Zhiyu An, and Wan Du from the University of California, Merced, propose a novel approach called Learning Progress Monitoring (LPM). Instead of rewarding prediction error or novelty, LPM rewards the agent for improvements in its understanding of the environment’s dynamics. This means the agent is encouraged to observe transitions that genuinely help it learn, rather than getting distracted by random, unlearnable events.
The core of LPM involves a dual-network design. One network, the dynamics model, predicts the next observation given the current state and action. The other, an error model, predicts the expected prediction error of the dynamics model from its previous iteration. The intrinsic reward is then calculated as the difference between the expected error of the previous model and the current model’s error. Essentially, if the model has significantly improved its prediction accuracy, the agent receives a positive intrinsic reward, guiding it towards more informative experiences.
The authors provide theoretical backing for LPM, demonstrating that its intrinsic reward is a reliable indicator of “Information Gain” – a measure of how much uncertainty about the model parameters is reduced after observing new data. This theoretical analysis also highlights why the dual-network design, particularly the error model, is crucial for maintaining this monotonic relationship with information gain.
Empirical evaluations compared LPM against state-of-the-art exploration methods in various noisy environments, including a noisy MNIST dataset, a 3D maze with complex visual inputs, and several Atari games. The results consistently showed LPM’s superior performance. In the noisy MNIST experiment, LPM’s intrinsic reward converged faster and was immediately robust to environmental stochasticity, unlike other methods that temporarily favored stochastic states. In the 3D maze, LPM explored more novel states and demonstrated remarkable robustness to both state and action noise, outperforming baselines that suffered significant degradation in noisy conditions.
Furthermore, in challenging Atari games with added noise, LPM achieved higher extrinsic rewards and maintained exceptional stability, experiencing minimal performance drops compared to other methods that showed substantial vulnerability to noise. This indicates that LPM provides a consistent and noise-robust exploration strategy, making it highly suitable for real-world environments where unpredictable noise is common. For more technical details, you can read the full paper here: BEYONDNOISY-TVS: NOISE-ROBUSTEXPLORATION VIALEARNINGPROGRESSMONITORING.
Also Read:
- TimeRewarder: A New Approach to Robotic Skill Acquisition Through Video Analysis
- Efficient Imitation Learning in Low-Data Scenarios: The Noise-Guided Transport Method
This conceptually simple yet powerful approach marks a significant shift in how noise-robust exploration can be achieved in reinforcement learning, moving beyond traditional prediction error or novelty-based intrinsic rewards to focus on the actual progress of an agent’s learning.
