The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspective

2025-01-27

AI Summary

This paper addresses the sample efficiency of online reinforcement learning for nonlinear dynamical systems with continuous state-action spaces, focusing on regret minimization for real-valued parametric policies (e.g., neural networks, Transformers). Methodologically, it introduces the first unified sample-complexity framework covering three broad model classes: finite model sets, Lipschitz dynamical systems, and compactly parameterized systems. The approach integrates optimistic exploration over model classes, packing-number-based function-class complexity characterization, confidence-set construction, and generalization-error control over compact parameter spaces. Key theoretical contributions include: (i) a general regret bound of $O(Nvarepsilon^2 + ln m(varepsilon)/varepsilon^2)$; (ii) the first extension of the optimal $O(sqrt{Np})$ regret rate—previously established only for linear systems—to nonlinear compactly parameterized systems; and (iii) an algorithm achieving interpretability, seamless incorporation of prior knowledge, and strong transient performance.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We study the sample complexity of online reinforcement learning for nonlinear dynamical systems with continuous state and action spaces. Our analysis accommodates a large class of dynamical systems ranging from a finite set of nonlinear candidate models to models with bounded and Lipschitz continuous dynamics, to systems that are parametrized by a compact and real-valued set of parameters. In the most general setting, our algorithm achieves a policy regret of $mathcal{O}(N epsilon^2 + mathrm{ln}(m(epsilon))/epsilon^2)$, where $N$ is the time horizon, $epsilon$ is a user-specified discretization width, and $m(epsilon)$ measures the complexity of the function class under consideration via its packing number. In the special case where the dynamics are parametrized by a compact and real-valued set of parameters (such as neural networks, transformers, etc.), we prove a policy regret of $mathcal{O}(sqrt{N p})$, where $p$ denotes the number of parameters, recovering earlier sample-complexity results that were derived for linear time-invariant dynamical systems. While this article focuses on characterizing sample complexity, the proposed algorithms are likely to be useful in practice, due to their simplicity, the ability to incorporate prior knowledge, and their benign transient behavior.

Problem

• Reinforcement Learning
• Sample Efficiency
• Continuous Action Spaces

Innovation

• Online Learning
• Nonlinear Dynamical Systems
• Regret Minimization