Le Critique: Privileged Value Functions for LLM Reinforcement Learning

2026-08-17

AI Summary

This study addresses the challenges of applying value functions in reinforcement learning for large language models and the inefficiency of Group Relative Policy Optimization (GRPO). We propose a Privileged Value Function coupled with a tethered adaptive interpolation mechanism to optimize credit assignment and enable dynamic baseline adjustment. This approach significantly enhances training stability and sample efficiency. Empirical results demonstrate that our method not only outperforms standard value function baselines in reasoning tasks but also matches or surpasses GRPO performance. By effectively overcoming existing bottlenecks, this work establishes an efficient new paradigm for optimizing the reasoning capabilities of large language models via reinforcement learning.

Abstract

Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.

Problem

• Large Language Models
• Reinforcement Learning
• Value Functions
• Credit Assignment
• Training Efficiency

Innovation

• Privileged Value Functions
• TETHER
• Token-level Credit Assignment
• LLM Reinforcement Learning
• Adaptive Baseline Interpolation