Computationally and Sample Efficient Safe Reinforcement Learning Using Adaptive Conformal Prediction

2025-03-22

AI Summary

To address safety assurance and uncertainty quantification in deploying learning-based autonomous systems under real-world conditions, this paper proposes a model-agnostic, sample- and computation-efficient online safe reinforcement learning framework. Methodologically, it innovatively integrates Adaptive Conformal Prediction (ACP) with Control Barrier Functions (CBFs) to construct uncertainty-aware safety constraints, and employs Quadrature Fourier Features (QFF)-accelerated Gaussian process dynamics modeling to balance real-time performance and theoretical rigor. Theoretically, we prove guaranteed constraint satisfaction and derive an upper bound on sample complexity. Empirical evaluation on nonlinear systems demonstrates 99.2% safe trajectory coverage, near-optimal cumulative reward, and a 67% reduction in inference latency compared to baseline approaches.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Large-scale security measurements

Abstract

Safety is a critical concern in learning-enabled autonomous systems especially when deploying these systems in real-world scenarios. An important challenge is accurately quantifying the uncertainty of unknown models to generate provably safe control policies that facilitate the gathering of informative data, thereby achieving both safe and optimal policies. Additionally, the selection of the data-driven model can significantly impact both the real-time implementation and the uncertainty quantification process. In this paper, we propose a provably sample efficient episodic safe learning framework that remains robust across various model choices with quantified uncertainty for online control tasks. Specifically, we first employ Quadrature Fourier Features (QFF) for kernel function approximation of Gaussian Processes (GPs) to enable efficient approximation of unknown dynamics. Then the Adaptive Conformal Prediction (ACP) is used to quantify the uncertainty from online observations and combined with the Control Barrier Functions (CBF) to characterize the uncertainty-aware safe control constraints under learned dynamics. Finally, an optimism-based exploration strategy is integrated with ACP-based CBFs for safe exploration and near-optimal safe nonlinear control. Theoretical proofs and simulation results are provided to demonstrate the effectiveness and efficiency of the proposed framework.

Problem

• Quantifying uncertainty for safe control policies
• Efficient safe learning with model robustness
• Safe exploration using adaptive conformal prediction

Innovation

• Quadrature Fourier Features for GP approximation
• Adaptive Conformal Prediction for uncertainty quantification
• Optimism-based exploration with CBFs for safe control