Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

2026-04-12

AI Summary

This work addresses the limitations of traditional discriminative value models in reinforcement learning with large language models, which often struggle with credit assignment due to restricted representational capacity. To overcome this, the authors propose the Generative Actor-Critic (GenAC) framework, replacing single-step scalar value prediction with a generative critic endowed with chain-of-thought reasoning capabilities. A context-conditioning mechanism is introduced to align the critic’s evaluations with the current policy. By reframing value modeling as a generative, sequential reasoning process rather than a discriminative prediction task, GenAC substantially enhances the expressiveness of the value function, improves ranking reliability, and boosts out-of-distribution generalization. Empirical results demonstrate consistent and significant improvements over both value-based and value-free baselines across downstream reinforcement learning tasks.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-grained advantage estimation based on a learned value function. However, learned value models are often avoided in modern large language model (LLM) RL because conventional discriminative critics are difficult to train reliably. We revisit value modeling and argue that this difficulty is partly due to limited expressiveness. In particular, representation complexity theory suggests that value functions can be hard to approximate under the one-shot prediction paradigm used by existing value models, and our scaling experiments show that such critics do not improve reliably with scale. Motivated by this observation, we propose Generative Actor-Critic (GenAC), which replaces one-shot scalar value prediction with a generative critic that performs chain-of-thought reasoning before producing a value estimate. We further introduce In-Context Conditioning, which helps the critic remain calibrated to the current actor throughout training. GenAC improves value approximation, ranking reliability, and out-of-distribution generalization, and these gains translate into stronger downstream RL performance than both value-based and value-free baselines. Overall, our results suggest that stronger value modeling is a promising direction for improving credit assignment in LLM reinforcement learning.

Problem

• credit assignment
• value modeling
• large language model
• reinforcement learning
• actor-critic

Innovation

• Generative Actor-Critic
• value modeling
• chain-of-thought reasoning
• In-Context Conditioning
• credit assignment