LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy

2026-02-19

AI Summary

This work addresses the issue of safety drift in offline reinforcement learning caused by the absence of structural mechanisms. To this end, the authors propose LexiSafe, a novel framework that, for the first time, integrates lexicographic prioritization with structural bias to prioritize safety constraints over task performance optimization using only static datasets. LexiSafe accommodates both single- and multi-safety-cost settings and offers theoretically provable guarantees on safety and sample complexity. Two concrete algorithms, LexiSafe-SC and LexiSafe-MC, are derived from this framework. Experimental results demonstrate that LexiSafe significantly reduces safety violations across multiple tasks while simultaneously improving task performance, outperforming existing constrained offline reinforcement learning approaches.

Technology Category

• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Offline safe reinforcement learning (RL) is increasingly important for cyber-physical systems (CPS), where safety violations during training are unacceptable and only pre-collected data are available. Existing offline safe RL methods typically balance reward-safety tradeoffs through constraint relaxation or joint optimization, but they often lack structural mechanisms to prevent safety drift. We propose LexiSafe, a lexicographic offline RL framework designed to preserve safety-aligned behavior. We first develop LexiSafe-SC, a single-cost formulation for standard offline safe RL, and derive safety-violation and performance-suboptimality bounds that together yield sample-complexity guarantees. We then extend the framework to hierarchical safety requirements with LexiSafe-MC, which supports multiple safety costs and admits its own sample-complexity analysis. Empirically, LexiSafe demonstrates reduced safety violations and improved task performance compared to constrained offline baselines. By unifying lexicographic prioritization with structural bias, LexiSafe offers a practical and theoretically grounded approach for safety-critical CPS decision-making.

Problem

• offline safe reinforcement learning
• safety drift
• cyber-physical systems
• lexicographic safety
• hierarchical safety constraints

Innovation

• lexicographic reinforcement learning
• offline safe RL
• safety drift prevention
• hierarchical safety constraints
• sample complexity guarantees