TraCeS: Trajectory Based Credit Assignment From Sparse Safety Feedback

2025-04-17

AI Summary

In safety-critical reinforcement learning, safety constraints are often unknown a priori, and only sparse, trajectory-level binary safety labels are available—hindering fine-grained safety credit assignment. Method: We propose a trajectory-based temporal credit assignment mechanism that, for the first time, infers the per-step contribution of each action to overall safety from global, sparse safety feedback. Our approach jointly learns an implicit safety definition via a trajectory encoder and a temporal-attention safety model, and integrates this into a safety-aware policy gradient framework compatible with continuous control. Contribution/Results: Evaluated on multiple benchmark tasks, our method achieves substantial improvements in safety compliance rate (+23.6% on average) while preserving high task performance, demonstrating strong generalization to unknown safety specifications and scalability to complex continuous-control domains.

Technology Category

• Natural Language Processing: Safety and Robustness
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In safe reinforcement learning (RL), auxiliary safety costs are used to align the agent to safe decision making. In practice, safety constraints, including cost functions and budgets, are unknown or hard to specify, as it requires anticipation of all possible unsafe behaviors. We therefore address a general setting where the true safety definition is unknown, and has to be learned from sparsely labeled data. Our key contributions are: first, we design a safety model that performs credit assignment to estimate each decision step's impact on the overall safety using a dataset of diverse trajectories and their corresponding binary safety labels (i.e., whether the corresponding trajectory is safe/unsafe). Second, we illustrate the architecture of our safety model to demonstrate its ability to learn a separate safety score for each timestep. Third, we reformulate the safe RL problem using the proposed safety model and derive an effective algorithm to optimize a safe yet rewarding policy. Finally, our empirical results corroborate our findings and show that this approach is effective in satisfying unknown safety definition, and scalable to various continuous control tasks.

Problem

• Estimating decision step impact on safety from sparse labels
• Learning separate safety scores for each timestep
• Optimizing policies for unknown safety constraints

Innovation

• Trajectory-based credit assignment for safety
• Separate safety score per timestep learning
• Reformulated safe RL with effective algorithm