reinforcement learning from human feedback

Existing RLHF methods erroneously assume human-labeled trajectories are generated by an optimal policy, leading to biased trajectory likelihood estimation and suboptimal policy learning. To address this, we propose Preference Learning with Policy Annotation (PPL), the first framework to explicitly model regret in preference learning—thereby recovering behavioral policy information and calibrating the trajectory distribution. Our key contributions are: (1) a regret-based preference likelihood model that mitigates likelihood misestimation; and (2) a derived contrastive KL regularization term that enhances policy stability and alignment in sequential decision-making. Evaluated on high-dimensional continuous control tasks, PPL significantly improves offline RLHF performance and demonstrates strong generalization and robustness in online RLHF settings.

This work addresses the mismatch between existing policy optimization methods and preference-based reward modeling in non-verifiable reward settings—such as summarization and instruction following—where reliance on absolute reward values limits performance. To resolve this, the authors propose Group-wise Ordinal Policy Optimization (GOPO), which, for the first time, directly incorporates ordinal ranking information from rewards into policy training. By applying ordinal transformations, GOPO aligns the objectives of preference learning and policy optimization while avoiding dependence on unreliable reward magnitudes. Experiments demonstrate that GOPO significantly improves training stability and sample efficiency across diverse tasks and model scales: compared to GRPO, it yields superior training and validation reward trajectories, achieves higher LLM-as-judge scores at most intermediate steps, and reaches comparable policy quality with fewer training iterations.

To address the high cost and complexity of human preference annotation in large language model (LLM) alignment, this paper proposes a novel inverse reinforcement learning (IRL) paradigm that relies solely on demonstration data. We theoretically establish that demonstration data implicitly encodes human preferences, thereby eliminating the need for explicit preference labeling in conventional RLHF and enabling end-to-end alignment. Methodologically, we introduce a joint optimization framework that simultaneously trains the policy and infers the reward function, ensuring compatibility with mainstream LLM architectures and open-source evaluation benchmarks. Experiments demonstrate that our approach achieves performance on par with or superior to current state-of-the-art demonstration-only methods across the Hugging Face Open LLM Leaderboard, MT-Bench, and public reward modeling benchmarks—while substantially reducing both data curation effort and engineering overhead.

To address the challenge of policy optimization under sentence-level sparse rewards in Reinforcement Learning from Human Feedback (RLHF), this paper proposes the Reward Token Optimization (RTO) algorithm. RTO is the first method to seamlessly integrate token-level quality representations implicitly learned by Direct Preference Optimization (DPO) with Proximal Policy Optimization (PPO), establishing a fine-grained Markov Decision Process (MDP)-based modeling framework that jointly trains token-level reward modeling and policy optimization. Theoretically, RTO achieves near-optimal trade-offs between sample efficiency and policy optimality. Empirically, RTO outperforms standard PPO by +7.5 and +4.1 points on AlpacaEval 2 and Arena-Hard, respectively, and significantly surpasses mainstream direct preference learning approaches. This work introduces an efficient, interpretable paradigm for aligning large language models under sparse-reward settings.

Large language models (LLMs) often internalize societal biases, hindering value alignment with human preferences. This paper addresses two key limitations of existing preference optimization methods: distributional shift in RLHF and insufficient robustness of DPO. We propose a two-stage hybrid preference optimization framework: first, preference data are stratified into “easy” and “hard” samples based on reward-gap thresholds; second, an initial policy is trained via DPO on the easy subset, then fine-tuned online via PPO-RLHF on the hard subset—using the DPO-trained policy as a dynamic reference model. To our knowledge, this is the first work to leverage DPO-trained policies as reference models in RLHF, establishing a synergistic paradigm that balances training efficiency and policy robustness. Extensive experiments on HH-RLHF and TLDR demonstrate significant improvements over state-of-the-art baselines. Both GPT-4-based automated evaluation and human assessment confirm that our method yields safer, more human-preferred outputs.

This paper addresses key challenges in constrained reinforcement learning (CRL) for safety-critical control, preference learning, and large language model (LLM) alignment. First, for average-cost and finite-horizon constrained Markov decision processes (CMDPs), we propose ACPO and e-COP—algorithms achieving both theoretical optimality and state-of-the-art constraint satisfaction. Second, for preference-based learning, we introduce warmPref-PS and PSPL, the first methods integrating uncertainty quantification with posterior sampling, significantly reducing regret and enhancing robust policy identification under sparse or noisy preferences. Third, for multi-objective LLM alignment, we develop MOPO—a scalable, constraint-aware optimization framework supporting billion-parameter models. All methods are underpinned by rigorous theoretical guarantees—including convergence, constraint violation bounds, and regret analysis—and empirically validated across diverse real-world benchmarks in robotics, human-AI interaction, and LLM fine-tuning.

In preference-based reward learning, reward models are vulnerable to “reward hacking”—exploiting spurious shortcuts (e.g., response length, overly polite phrasing) rather than aligning with true human intent, leading to poor out-of-distribution generalization. This work is the first to formally unify such failures under the “shortcut behavior” problem. We propose PRISM, a principled framework grounded in kernel invariant learning: it constructs group-invariant kernel functions and feature mappings for preference data and solves for invariant reward modeling via closed-form optimization, systematically suppressing diverse shortcut dependencies. Experiments demonstrate that PRISM significantly improves reward model accuracy on out-of-distribution preferences and reduces downstream policy models’ reliance on spurious features—validating its robustness and generalization capability across diverse benchmarks.

In reinforcement learning, hand-crafted reward functions often misalign with true human objectives, leading to reward hacking. To address this, we propose a preference-based reward repair framework: given a small set of pairwise preference labels over state transitions, it iteratively optimizes a learnable, transition-specific additive correction term that fuses prior reward signals with human feedback; concurrently, a directed exploration strategy prioritizes correction at critical transitions. Theoretically, our method achieves a regret bound matching the state-of-the-art for comparable approaches. Empirically, we evaluate on multiple reward-hacking benchmarks and demonstrate that—using only a handful of preference annotations—our method significantly outperforms both from-scratch policy training and conventional reward redesign techniques, efficiently recovering near-optimal policy performance.

DPO suffers from model misspecification when the policy class cannot represent the true reward function, leading to preference reversal, policy degradation, and sensitivity to preference distribution. To address this, we propose AuxDPO: a method that introduces auxiliary variables to correct bias, models natural gradient updates in RLHF from a geometric perspective, and reformulates the loss function within a statistical estimation and supervised learning framework. Its core innovation lies in explicitly modeling the implicit reward function—thereby alleviating DPO’s strong dependence on the expressive capacity of the policy class. Experiments on teaching bandits and large language model alignment tasks demonstrate that AuxDPO consistently outperforms standard DPO, achieving superior robustness under preference noise. These results empirically validate AuxDPO’s effectiveness in mitigating model misspecification.

Existing interactive imitation learning corrects only the agent’s immediate behavior upon human intervention, neglecting proactive avoidance of future high-risk states. To address this, we propose a predictive preference learning framework that leverages implicit preference signals embedded in human interventions to forecast behavioral risk over an L-step lookahead horizon. It actively propagates each intervention to potentially hazardous trajectory segments via temporal preference propagation, intervention-guided trajectory prediction, and preference-aware policy optimization—thereby generalizing expert corrections to safety-critical regions. Theoretical analysis guides the optimal selection of the lookahead horizon L, balancing label fidelity and coverage. Experiments on autonomous driving and robotic manipulation tasks demonstrate that our method significantly reduces human interventions (by 37% on average), enhances policy safety and sample efficiency, and validates its generality and effectiveness across multiple benchmarks.