Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control

2026-03-11

AI Summary

This work addresses the limitations of existing reward learning from human feedback (RLHF) methods that rely on expected cost constraints, which fail to account for tail risks and distributional uncertainty, thereby compromising safety in the face of rare catastrophic events. To overcome this, the authors propose Risk-sensitive Alignment via Dominance (RAD), a novel framework that introduces first-order stochastic dominance (FSD) into RLHF for safe policy alignment. RAD compares the full cost distributions of the target and reference policies via optimal transport and establishes a theoretical connection between weighted FSD and spectral risk measures, enabling flexible control over risk preferences. By integrating entropy regularization and Sinkhorn iterations, RAD enables differentiable and computationally efficient optimization under FSD constraints. Experiments demonstrate that RAD significantly enhances harmlessness while preserving helpfulness and exhibits superior robustness over existing baselines in out-of-distribution scenarios.

Technology Category

• Natural Language Processing: Safety and Robustness
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncertainty, particularly under heavy tails or rare catastrophic events. This limitation is problematic when robustness and risk sensitivity are critical. Stochastic dominance offers a principled alternative by comparing entire cost distributions rather than just their averages, enabling direct control over tail risks and potential out-of-distribution failures that expectation-based constraints may overlook. In this work, we propose Risk-sensitive Alignment via Dominance (RAD), a novel alignment framework that replaces scalar expected cost constraints with First-Order Stochastic Dominance (FSD) constraints. We operationalize this constraint by comparing the target policy's cost distribution to that of a reference policy within an Optimal Transport (OT) framework, using entropic regularization and Sinkhorn iterations to obtain a differentiable and computationally efficient objective for stable end-to-end optimization. Furthermore, we introduce quantile-weighted FSD constraints and show that weighted FSD universally controls a broad class of Spectral Risk Measures (SRMs), so that improvements under weighted dominance imply guaranteed improvements in the corresponding spectral risk. This provides a principled mechanism for tuning a model's risk profile via the quantile weighting function. Empirical results demonstrate that RAD improves harmlessness over baselines while remaining competitive in helpfulness, and exhibits greater robustness on out-of-distribution harmlessness evaluations.

Problem

• Safe Reinforcement Learning
• Human Feedback
• Stochastic Dominance
• Risk Sensitivity
• Spectral Risk Measures

Innovation

• Stochastic Dominance
• Spectral Risk Measures
• Optimal Transport
• Safe RLHF
• Risk-sensitive Alignment