Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation

2024-05-31

International Conference on Machine Learning

AI Summary

This work addresses the instability of value estimation in off-policy bootstrapping caused by function approximation, focusing on the synergistic interplay between target networks and overparameterized linear function approximators. Theoretically, we provide the first rigorous proof that their joint use is necessary to ensure convergence under specific off-policy settings, and derive a high-probability upper bound on the value estimation error. Methodologically, we extend the framework to truncated trajectories, enabling stable learning for general tasks. Experiments on the Baird counterexample and the Four-room domain demonstrate that this combination significantly improves training stability and convergence robustness compared to baselines. Our results offer critical theoretical grounding for off-policy deep reinforcement learning—particularly DQN-style algorithms—and elucidate the fundamental role of target networks in overparameterized regimes.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We prove that the combination of a target network and over-parameterized linear function approximation establishes a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data. Our condition is naturally satisfied for expected updates over the entire state-action space or learning with a batch of complete trajectories from episodic Markov decision processes. Notably, using only a target network or an over-parameterized model does not provide such a convergence guarantee. Additionally, we extend our results to learning with truncated trajectories, showing that convergence is achievable for all tasks with minor modifications, akin to value truncation for the final states in trajectories. Our primary result focuses on temporal difference estimation for prediction, providing high-probability value estimation error bounds and empirical analysis on Baird's counterexample and a Four-room task. Furthermore, we explore the control setting, demonstrating that similar convergence conditions apply to Q-learning.

Problem

• Stabilizes off-policy bootstrapping with target networks and over-parameterization
• Establishes convergence conditions for temporal difference value estimation
• Extends convergence guarantees to truncated trajectory learning scenarios

Innovation

• Target networks stabilize off-policy bootstrapping with function approximation
• Over-parameterized linear models enable convergence with target networks
• Combined approach ensures convergence for truncated trajectory learning