ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization

2026-05-12

AI Summary

This work addresses the instability in large language model (LLM) alignment caused by the stochasticity of discrete rewards generated by LLM-based automatic scorers, which can corrupt advantage estimation and hinder policy optimization. To mitigate this issue, the authors propose the Ordinal Decomposition Reward-based Policy Optimization (ODRPO) framework, which decomposes discrete rewards into ordinal binary indicators and independently computes and accumulates advantages across multiple success thresholds. This approach isolates evaluation noise through ordinal decomposition and implicitly achieves variance-aware learning without incurring additional computational overhead. Experiments on Qwen2.5-7B and Qwen3-4B demonstrate relative performance improvements of up to 14.8% and 7.5%, respectively, with significantly enhanced training stability and efficiency compared to baseline methods.

Abstract

The alignment of Large Language Models (LLMs) utilizes Reinforcement Learning from AI Feedback (RLAIF) for non-verifiable domains such as long-form question answering and open-ended instruction following. These domains often rely on LLM based auto-raters to provide granular, multi-tier discrete rewards (e.g., 1-10 rubrics) that are inherently stochastic due to prompt sensitivity and sampling randomness. We empirically verify the stochasticity of auto-raters that can propagate and corrupt standard advantage estimators like GRPO and MaxRL, as a noisy reward samples can skew normalization statistics and degrade the global learning signal. Empirically, sampling more rewards and taking majority voting may reduce the noise and improve performance, but this approach is computationally expensive. To address this bottleneck, we introduce $\textbf{O}$rdinal $\textbf{D}$ecomposition for $\textbf{R}$obust $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{ODRPO}$), a framework that structurally isolates evaluation noise by decomposing discrete rewards into a sequence of ordinal binary indicators. By independently computing and accumulating advantages across these progressively challenging success thresholds, ODRPO prevents outlier evaluations from corrupting the global update while establishing an implicit, variance-aware learning curriculum. Empirically, ODRPO achieves robust performance on Qwen2.5-7B and Qwen3-4B models, outperforming baselines with relative improvements of upto 14.8% on FACTS-grounding-v2 and 7.5% on Alpaca-Evals. Critically, these gains are achieved with negligible training-time overhead, as ODRPO requires no additional compute per step compared to standard estimators. Supported by theoretical analysis confirming its optimization stability, ODRPO provides a scalable and robust framework for aligning models within the noisy, discrete evaluation landscape of modern RLAIF.

Problem

• discrete rewards
• stochasticity
• RLAIF
• advantage estimation
• evaluation noise

Innovation

• Ordinal Decomposition
• Robust Policy Optimization
• Discrete Rewards
• RLAIF
• Advantage Estimation