The Formalism-Implementation Gap in Reinforcement Learning Research

2025-10-17

AI Summary

This paper identifies a “formalism–implementation gap” in reinforcement learning (RL) research: an overemphasis on agent performance on benchmarks (e.g., the Arcade Learning Environment, ALE), at the expense of scientific understanding of learning dynamics and mechanisms—leading to benchmark overfitting, theory–practice disconnect, and poor generalizability. To address this, the authors systematically analyze mismatches between standard implementations of mainstream RL algorithms in ALE and the formal Markov Decision Process (MDP) assumptions underlying RL theory—revealing implicit violations of theoretical premises via engineering practices such as reward shaping, state representation design, and action masking. Their core contribution is the “formalism–benchmark alignment” principle, advocating a comprehension-driven research paradigm: benchmark design must be grounded in mathematical fundamentals to ensure experimental reproducibility, mechanistic interpretability, and theoretical generalizability. Empirical analysis demonstrates that even “saturated” benchmarks retain significant value for deep mechanistic inquiry.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

The last decade has seen an upswing in interest and adoption of reinforcement learning (RL) techniques, in large part due to its demonstrated capabilities at performing certain tasks at "super-human levels". This has incentivized the community to prioritize research that demonstrates RL agent performance, often at the expense of research aimed at understanding their learning dynamics. Performance-focused research runs the risk of overfitting on academic benchmarks -- thereby rendering them less useful -- which can make it difficult to transfer proposed techniques to novel problems. Further, it implicitly diminishes work that does not push the performance-frontier, but aims at improving our understanding of these techniques. This paper argues two points: (i) RL research should stop focusing solely on demonstrating agent capabilities, and focus more on advancing the science and understanding of reinforcement learning; and (ii) we need to be more precise on how our benchmarks map to the underlying mathematical formalisms. We use the popular Arcade Learning Environment (ALE; Bellemare et al., 2013) as an example of a benchmark that, despite being increasingly considered "saturated", can be effectively used for developing this understanding, and facilitating the deployment of RL techniques in impactful real-world problems.

Problem

• Addresses overfitting on academic benchmarks in RL research
• Bridges the gap between RL formalism and implementation practices
• Advocates shifting focus from performance to understanding learning dynamics

Innovation

• Focus on understanding reinforcement learning dynamics
• Precise mapping of benchmarks to mathematical formalisms
• Use saturated benchmarks for developing scientific insights