Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning

2026-09-10

AI Summary

论文提出一种基于片段比较和偶尔询问是否超出预算的方法，通过过滤-克隆流程确保离线强化学习的安全性，并提供无分布保证。

Abstract

Safe offline reinforcement learning assumes a cost function on every transition. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget. Certified safety curation answers with a filter-then-clone pipeline: a state-only value trained from segment comparisons scores whole trajectories, Learn-then-Test calibration certifies a selection threshold under a distribution-free $(α, δ)$ bound on the unsafe fraction of the selection, and behavior cloning follows. We are not aware of prior work certifying the composition of a training set for offline RL or imitation. Oracle controls justify the design: reweighting individual transitions fails even with an exact value, so the value selects whole trajectories. The policies satisfy the cost budget on eleven of fifteen DSRL tasks, one short of cloning the ground-truth safe subset, which needs a label on every trajectory; the uncertified variant reaches twelve. Retrained on the certified selection, the strongest full-label method becomes safe where no setting of its own cost target rescues it. Refusal is predictable: the certificate's probability has a closed form in the purity the pool attains, which the calibration sample estimates and the scorer enters only through.

Problem

• Safe Offline Reinforcement Learning
• Distribution-Free Guarantees
• Cost Function
• Trajectory Safety
• Certified Safety

Innovation

• Certified Safety Curation
• Distribution-Free Guarantees
• Offline Reinforcement Learning
• Filter-Then-Clone Pipeline
• Learn-Then-Test Calibration