TLDR: This paper argues that Reinforcement Learning (RL) research should move beyond solely demonstrating agent performance on benchmarks like the Arcade Learning Environment (ALE) and instead prioritize scientific understanding of learning dynamics. It highlights a “formalism-implementation gap,” where mathematical definitions of RL problems don’t precisely map to practical implementations and benchmark evaluations. The paper recommends being explicit about these mappings, consistently reporting evaluation metrics, and focusing on per-game analysis rather than aggregate scores to foster more transferable and robust RL techniques.
Reinforcement Learning (RL) has seen a significant surge in interest over the last decade, largely due to its impressive ability to achieve ‘super-human’ performance in various tasks. However, this focus on performance has inadvertently led to a gap in our understanding of how these RL agents truly learn and operate. A recent paper, “The Formalism-Implementation Gap in Reinforcement Learning Research”, argues for a crucial shift in the RL research paradigm: moving away from solely demonstrating agent capabilities towards a deeper scientific understanding of the field.
The authors, led by Pablo Samuel Castro from Google DeepMind, Université de Montréal, and Mila – Québec AI Institute, highlight two main points. Firstly, RL research needs to prioritize advancing the science and understanding of reinforcement learning, rather than just showcasing agent performance. Secondly, there’s a need for more precise mapping between the mathematical formalisms that define RL problems and the benchmarks used to evaluate them.
The paper uses the popular Arcade Learning Environment (ALE) as a prime example. Despite being considered ‘saturated’ by many, the ALE can still be a valuable tool for developing a robust understanding of RL techniques and facilitating their deployment in real-world problems. The core issue, termed the ‘formalism-implementation gap,’ arises because RL algorithms are typically defined abstractly using mathematical concepts like Markov Decision Processes (MDPs), but their practical implementations involve numerous design choices and hyper-parameters that are not always explicitly linked back to the underlying theory.
The Hidden Impact of Implementation Choices
The paper meticulously details how various implementation choices in benchmarks like the ALE can significantly alter the problem an RL agent is actually solving, often without researchers being fully explicit about these changes. For instance:
- State Space: Options like ‘frame skipping’ (repeating actions multiple times) and ‘frame stacking’ (combining multiple past frames as input) dramatically affect what an agent perceives as its ‘state.’ This often means agents are not operating in a purely Markovian fashion, effectively turning the ALE into a Partially-Observable MDP (POMDP) in practice.
- Action Space: The use of ‘minimal action sets’ (reducing the number of available actions) or the recent introduction of ‘continuous actions’ can change game difficulty and consistency across different environments.
- Initial State Distribution: Adding ‘no-op’ actions at the start of an episode, while intended to prevent overfitting, introduces a non-trivial and more complex starting state distribution than the original deterministic system.
- Reward Function: ‘Reward clipping’ (limiting rewards to a range like [-1, 1]) simplifies numerical optimization but can obscure the true magnitude of rewards, leading to partially observable rewards.
- Environment Dynamics: Decisions like whether ‘life-loss events’ trigger an episode termination can significantly impact how an agent learns, and these defaults can even vary between popular RL libraries.
Rethinking How We Measure Progress
Beyond implementation details, the paper also scrutinizes how progress is measured in RL. It argues that current evaluation procedures can dramatically affect conclusions and limit the generality of findings:
- Discount Factor (γ): This crucial parameter, central to most RL algorithms, is often not explicitly reported in performance metrics, which typically focus on undiscounted cumulative rewards.
- Experiment Length: The duration of training can significantly influence the ranking of algorithms, yet there’s no consistent standard, often driven by computational constraints rather than scientific questions.
- Choice of Training and Evaluation Environments: Researchers often use overlapping or varied subsets of games for training and evaluation, making consistent comparisons difficult.
- Aggregate Results: Relying solely on aggregate performance metrics across many games can mask important per-game sensitivities to hyper-parameters, leading to misleading conclusions about an algorithm’s true characteristics. The paper advocates for more fine-grained, per-game analyses.
Also Read:
- Bridging the Reality Gap: A New Approach to Sim2Real Reinforcement Learning
- Unlocking Complex Rewards: A New Approach to Reinforcement Learning with RML Reward Machines
Desiderata for Better Benchmarks
To foster more robust and transferable RL techniques, the paper outlines key characteristics for effective benchmarks:
- Well Understood: Benchmarks should be familiar and their nuances easily grasped by a diverse research community.
- Diverse and Unbiased: Environment suites should offer variety and ideally be developed independently of RL objectives to avoid ‘reward hacking’ or experimenter bias. The ALE, designed for human enjoyment, fits this criterion well.
- Naturally Extendable: Benchmarks should allow for novel research without requiring entirely new environments, enabling exploration of new agent features (e.g., masked observations, object-centric variants, multiplayer support, continuous actions).
In conclusion, the paper urges the RL community to move beyond ‘SotA-chasing’ (State-of-the-Art chasing) and aggregate metrics. Instead, it calls for a focus on developing scientific insights through well-specified evaluation benchmarks, explicit formalism-implementation mappings, and detailed per-game analyses. This shift, the authors contend, will lead to more reliable, transferable, and ultimately more useful RL algorithms for addressing impactful real-world problems.
