Replicable Reinforcement Learning with Linear Function Approximation

2025-09-10

AI Summary

This work addresses the fundamental reproducibility challenge in reinforcement learning (RL) under function approximation—where algorithmic results are notoriously difficult to replicate. For the first time, it extends formal reproducibility guarantees from tabular Markov decision processes (MDPs) to the linear function approximation setting. We propose a reproducible RL framework grounded in linear MDPs, integrating stochastic design regression, non-centered covariance estimation, and derandomization techniques. The framework provides rigorous theoretical guarantees for both generative and episodic RL settings. Our algorithm achieves a principled trade-off between computational efficiency and output consistency, with theoretical proofs establishing both reproducibility and near-optimal sample complexity. Empirical evaluations demonstrate substantial improvements in the consistency of neural policy training across independent runs. This work delivers the first linear-function-approximation RL method with provable reproducibility and strong theoretical foundations, bridging a critical gap toward practical, reliable deployment of RL systems.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Replication of experimental results has been a challenge faced by many scientific disciplines, including the field of machine learning. Recent work on the theory of machine learning has formalized replicability as the demand that an algorithm produce identical outcomes when executed twice on different samples from the same distribution. Provably replicable algorithms are especially interesting for reinforcement learning (RL), where algorithms are known to be unstable in practice. While replicable algorithms exist for tabular RL settings, extending these guarantees to more practical function approximation settings has remained an open problem. In this work, we make progress by developing replicable methods for linear function approximation in RL. We first introduce two efficient algorithms for replicable random design regression and uncentered covariance estimation, each of independent interest. We then leverage these tools to provide the first provably efficient replicable RL algorithms for linear Markov decision processes in both the generative model and episodic settings. Finally, we evaluate our algorithms experimentally and show how they can inspire more consistent neural policies.

Problem

• Develop replicable reinforcement learning with linear function approximation
• Address algorithm instability in reinforcement learning through provable replicability
• Extend replicability guarantees from tabular to function approximation settings

Innovation

• Replicable linear function approximation for RL
• Efficient algorithms for replicable regression and covariance estimation
• Provably efficient replicable RL for linear MDPs