A Measure-Theoretic Finite-Sample Theory for Adaptive-Data Fitted Q-Iteration

2026-05-07

AI Summary

This work addresses the lack of finite-sample theoretical guarantees for model-free off-policy deep reinforcement learning in complex nonlinear systems by establishing a unified theoretical framework that integrates measure-theoretic Markov decision processes, contraction analysis of Bellman operators, and sequential Rademacher complexity. Within this framework, the paper provides the first path-dependent finite-sample performance bound for fitted Q-iteration (FQI) algorithms over general Borel spaces and derives the first cumulative online regret bound applicable to continuous state spaces. These results unify three major theoretical strands—error propagation, PAC generalization, and adaptive data collection—thereby laying a rigorous theoretical foundation for modern deep reinforcement learning algorithms.

Abstract

While reinforcement learning (RL) promises to revolutionize the control of complex nonlinear robotic systems, a profound gap persists between the heuristic success of model-free off-policy deep RL and the underlying theory, which remains largely confined to tabular or linearizable settings. We identify the cause of this gap as an emergent isolation of three traditions: (i) measure-theoretic MDP foundations on general spaces limit their analysis to exact dynamic programming and ignore all error sources of a learning process; (ii) deterministic error propagation analysis addresses the approximation error via concentrability coefficients without a finite-sample analysis of the estimation error; and (iii) PAC generalization bounds characterize the estimation errors of simplified topologies. We bridge these traditions with a unified theoretical framework for fitted Q-iteration (FQI) on general measurable Borel spaces. Our main result provides a finite-sample, adaptive-data performance bound by chaining measure-theoretic probability with Bellman-operator contraction in Banach spaces. We prove that sequential Rademacher complexity controls Bellman-regression generalization under policy-dependent data collection. We further extend this analysis to provide the first cumulative, pathwise online regret guarantee for FQI in continuous spaces. These results lay the necessary foundations for the formal analysis of many modern deep RL algorithms.

Problem

• reinforcement learning
• fitted Q-iteration
• finite-sample theory
• measure-theoretic MDP
• adaptive-data

Innovation

• measure-theoretic RL
• fitted Q-iteration
• finite-sample analysis
• sequential Rademacher complexity
• online regret bound