Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dual

2026-02-25

AI Summary

This work addresses the challenges of instability and lack of last-iterate convergence guarantees in existing reinforcement learning from human feedback (RLHF) methods under parameterized policies, particularly in multi-objective safety alignment settings. The authors propose a unified primal-dual framework that encompasses mainstream safe alignment algorithms and introduce an Optimistic Primal-Dual (OPD) algorithm, which stabilizes saddle-point dynamics through predictive updates. For the first time, the paper establishes last-iterate convergence guarantees for safe large language model (LLM) alignment under parameterized policies, highlighting the critical role of optimism in mitigating oscillations between competing objectives and bridging the gap between theory and practice. Under exact optimization, last-iterate convergence is rigorously proven; under parameterized settings, the method converges to a neighborhood of the optimal solution, with error bounds determined by approximation errors and estimation bias, thereby significantly enhancing training stability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Reinforcement Learning from Human Feedback (RLHF) plays a significant role in aligning Large Language Models (LLMs) with human preferences. While RLHF with expected reward constraints can be formulated as a primal-dual optimization problem, standard primal-dual methods only guarantee convergence with a distributional policy where the saddle-point problem is in convex-concave form. Moreover, standard primal-dual methods may exhibit instability or divergence in the last iterate under policy parameterization in practical applications. In this work, we propose a universal primal-dual framework for safe RLHF that unifies a broad class of existing alignment algorithms, including safe-RLHF, one-shot, and multi-shot based methods. Building on this framework, we introduce an optimistic primal-dual (OPD) algorithm that incorporates predictive updates for both primal and dual variables to stabilize saddle-point dynamics. We establish last-iterate convergence guarantees for the proposed method, covering both exact policy optimization in the distributional space and convergence to a neighborhood of the optimal solution whose gap is related to approximation error and bias under parameterized policies. Our analysis reveals that optimism plays a crucial role in mitigating oscillations inherent to constrained alignment objectives, thereby closing a key theoretical gap between constrained RL and practical RLHF.

Problem

• Last-Iterate Convergence
• Safe RLHF
• Primal-Dual Optimization
• Constrained Alignment
• Large Language Models

Innovation

• Optimistic Primal-Dual
• Last-Iterate Convergence
• Safe RLHF
• Constrained Alignment
• Saddle-Point Dynamics