Redistribution-based Cost Inference Improves Sparse Safe Offline RL

2026-08-12

AI Summary

This work addresses the challenge in safe offline reinforcement learning where only sparse, trajectory-level termination feedback is available, as opposed to dense step-wise cost annotations typically assumed. Framing this as a temporal credit assignment problem, the paper introduces a lossless reward-equivalent redistribution mechanism that decomposes trajectory-level feedback into step-wise costs. This transformation theoretically preserves both the feasible policy set and the optimal Lagrangian of the underlying constrained Markov decision process (CMDP), while improving the learning conditions for the cost critic. Built upon this framework and integrated with offline policy optimization, the approach significantly reduces constraint violations in highway driving and robotic manipulation tasks, outperforming existing sparse-feedback and classifier-based baselines, and demonstrates strong robustness to data heterogeneity and label noise.

Abstract

Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors provide only trajectory-level stop-feedback: a binary signal at the first unsafe transition, with no per-step attribution. We frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which converts sparse stop-feedback into dense per-step costs via return decomposition, then trains a constrained offline policy on the augmented dataset. We show that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation demonstrate substantially lower violation rates than sparse and classifier-based baselines, with robustness to heterogeneous dataset compositions and label noise.

Problem

• safe offline reinforcement learning
• sparse feedback
• temporal credit assignment
• cost inference
• constrained MDP

Innovation

• Redistribution-based Cost Inference
• Safe Offline Reinforcement Learning
• Temporal Credit Assignment
• Constrained Markov Decision Process
• Return Decomposition