Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

2026-04-20

AI Summary

This work addresses a critical limitation in existing safe reinforcement learning methods based on human feedback, which typically rely on fixed-length trajectories and reward models fitted to human preferences, thereby lacking theoretical guarantees for infinite-horizon interactive settings. The authors formulate the problem as an infinite-horizon discounted constrained Markov decision process (CMDP) and propose two reward-model-free policy gradient primal-dual algorithms that accommodate trajectories of arbitrary length. As the first study to analyze infinite-horizon discounted CMDPs under human feedback, this paper establishes global non-asymptotic convergence guarantees, achieving polynomial convergence rates with respect to the number of policy updates, trajectory length, and the number of human preference queries.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Safe Reinforcement Learning from Human Feedback (Safe RLHF) has recently achieved empirical success in developing helpful and harmless large language models by decoupling human preferences regarding helpfulness and harmlessness. Existing approaches typically rely on fitting fixed horizon reward models from human feedback and have only been validated empirically. In this paper, we formulate safe RLHF as an infinite horizon discounted Con- strained Markov Decision Process (CMDP), since humans may interact with the model over a continuing sequence of interactions rather than within a single finite episode. We propose two Safe RLHF algorithms that do not require reward model fitting and, in contrast to prior work assuming fixed-length trajectories, support flexible trajectory lengths for training. Both algo- rithms are based on the primal-dual method and achieve global convergence guarantees with polynomial rates in terms of policy gradient iterations, trajectory sample lengths, and human preference queries. To the best of our knowledge, this is the first work to study infinite horizon discounted CMDP under human feedback and establish global, non-asymptotic convergence.

Problem

• Safe Reinforcement Learning
• Human Feedback
• Constrained Markov Decision Process
• Infinite Horizon
• Policy Gradient

Innovation

• Safe RLHF
• Constrained MDP
• primal-dual method
• infinite horizon
• global convergence