DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training

2025-12-03

AI Summary

To address policy instability and poor generalization in large language model (LLM) post-training caused by noisy or incomplete reinforcement learning (RL) supervision, this paper proposes a distributed risk-aware RL framework. Methodologically, it introduces Conditional Value-at-Risk (CVaR) theory into token-level distributional value modeling for the first time, and designs an asymmetric risk regularization: contracting the lower tail to suppress noise-induced deviations while preserving the upper tail to retain exploratory diversity. This balances robustness against over-conservatism, thereby enhancing policy generalization. Experiments across multi-turn dialogue, mathematical reasoning, and scientific question answering demonstrate that our method consistently outperforms PPO, GRPO, and robust Bellman-PPO under noisy supervision—achieving superior stability and cross-task transferability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Reinforcement learning (RL) has shown strong performance in LLM post-training, but real-world deployment often involves noisy or incomplete supervision. In such settings, complex and unreliable supervision signals can destabilize training and harm generalization. While existing approaches such as worst-case optimization (e.g., RFQI, CQL) and mean-based methods (e.g., PPO, GRPO) can improve stability, they often overlook generalization and may produce overly conservative policies, leading to uneven performance across diverse real scenarios. To this end, we introduce DVPO (Distributional Value Modeling with Risk-aware Policy Optimization), a new RL framework that combines conditional risk theory with distributional value modeling to better balance robustness and generalization. DVPO learns token-level value distributions to provide fine-grained supervision, and applies an asymmetric risk regularization to shape the distribution tails: it contracts the lower tail to dampen noisy negative deviations, while expanding the upper tail to preserve exploratory diversity. Across extensive experiments and analysis in multi-turn dialogue, math reasoning, and scientific QA, DVPO consistently outperforms PPO, GRPO, and robust Bellman-based PPO under noisy supervision, showing its potential for LLM post-training in the real-world.

Problem

• Address noisy supervision in LLM post-training
• Balance robustness and generalization in RL
• Improve policy performance across diverse scenarios

Innovation

• Distributional value modeling for token-level supervision
• Asymmetric risk regularization on distribution tails
• Conditional risk theory for robustness-generalization balance