A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions

2026-06-15

AI Summary

This work addresses the absence of a unified principled framework in existing large language model policy optimization methods, which obscures the mechanistic roles and design motivations of diverse algorithms within their objective functions. Starting from the expected reward objective, the paper constructs a diagnostic unified framework structured around two axes: trajectory-side and reward-side factors—centered on trajectory probabilities and rewards, respectively. Through first-principles derivation, this framework systematically integrates the evolutionary logic of REINFORCE, PPO, GRPO, and their variants (e.g., Agentic RL, GRPO-OPD), revealing compound failure modes that cannot be resolved by improvements on either axis alone and delineating the failure boundaries of current approaches. The proposed framework provides a principled foundation for joint policy optimization, offering both extensibility and diagnostic capability.

Abstract

Policy gradient algorithms for language models optimize the same objective $J(θ) = \mathbb{E}*{τ\sim p*θ(τ)}[R(τ)]$, which has exactly two factors: the trajectory probability $p_θ(τ)$ and the reward $R(τ)$. Every method from REINFORCE to PPO to GRPO and their descendants modifies one or both factors to address a specific failure in the preceding formulation. Existing surveys organize these methods by domain or chronology, which obscures the rationale behind each design choice and the precise location of its intervention within the gradient estimator. This survey revisits the landscape of LLM policy optimization from $J(θ)$ on first principles and uses the trajectory side, induced by $p_θ(τ)$, and the reward side, induced by $R(τ)$, as the two axes along which methods are located. It covers the path from REINFORCE and PPO to GRPO, as well as post-GRPO variants, Agentic RL, and GRPO-OPD. The resulting framework is unified, diagnostic, and extensible: it analyzes methods from a shared objective, identifies which side each method modifies and why, and applies the same trajectory and reward axes across these settings. Across these settings, the framework also exposes compound failures that no single-side fix resolves and that therefore require joint design of the trajectory side and the reward side. The boundary cases and coupled failures identified by this map mark where existing solutions run out and provide a principled starting point for designing the next generation of LLM policy optimization algorithms.

Problem

• LLM policy optimization
• expected reward
• trajectory probability
• reward function
• policy gradient

Innovation

• first-principles derivation
• trajectory-reward decomposition
• LLM policy optimization
• GRPO
• unified framework