NeoRL: Efficient Exploration for Nonepisodic RL

2024-06-03

Neural Information Processing Systems

AI Summary

This work addresses single-trajectory (no-reset) reinforcement learning for unknown nonlinear dynamical systems, where the agent observes only one continuous state-action sequence without resampling or system restarts. To tackle insufficient exploration and challenging policy optimization in non-episodic settings, we propose an optimistic planning framework grounded in Gaussian process (GP) dynamics modeling. Our contributions are threefold: (i) we derive the first $O(Gamma_T sqrt{T})$ cumulative regret upper bound for general nonlinear GP-based dynamics; (ii) we introduce a cognitive uncertainty calibration mechanism that balances conservative incentive with safe exploration; and (iii) we integrate energy-based stability analysis for continuous-time systems to ensure closed-loop Lyapunov stability. Evaluated on multiple deep RL benchmarks, our method achieves significantly lower cumulative regret and optimal average cost, outperforming existing single-trajectory RL baselines.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study the problem of nonepisodic reinforcement learning (RL) for nonlinear dynamical systems, where the system dynamics are unknown and the RL agent has to learn from a single trajectory, i.e., without resets. We propose Nonepisodic Optimistic RL (NeoRL), an approach based on the principle of optimism in the face of uncertainty. NeoRL uses well-calibrated probabilistic models and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics. Under continuity and bounded energy assumptions on the system, we provide a first-of-its-kind regret bound of $O(Gamma_T sqrt{T})$ for general nonlinear systems with Gaussian process dynamics. We compare NeoRL to other baselines on several deep RL environments and empirically demonstrate that NeoRL achieves the optimal average cost while incurring the least regret.

Problem

• Nonepisodic reinforcement learning
• Unknown nonlinear dynamics
• Single trajectory learning

Innovation

• Optimistic nonepisodic reinforcement learning
• Calibrated probabilistic models
• Gaussian process dynamics