DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training

2026-02-05

AI Summary

This work addresses the instability of reinforcement learning in large language model post-training, which often stems from noisy supervision signals and poor out-of-distribution generalization. To mitigate these issues, the authors propose DFPO, a novel framework that extends value modeling from discrete quantiles to a continuous value flow field across time steps. DFPO incorporates trajectory-wise conditional risk control and consistency constraints to enable more precise state-dependent value estimation. Empirical results demonstrate that DFPO significantly outperforms baseline methods such as PPO and FlowRL across dialogue, mathematical reasoning, and scientific tasks, exhibiting superior training stability and generalization robustness under both noisy supervision and out-of-distribution evaluation scenarios.

Technology Category

• Machine Learning: Reinforcement Learning
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain (OOD) generalization, especially in LLM post-training. Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar. This results in rough-grained value representations that lack fine-grained conditioning on state information, struggling under complex and OOD conditions. We propose DFPO (Distributional Value Flow Policy Optimization with Conditional Risk and Consistency Control), a robust distributional RL framework that models values as continuous flows across time steps. By scaling value modeling through learning of a value flow field instead of isolated quantile predictions, DFPO captures richer state information for more accurate advantage estimation. To stabilize training under noisy feedback, DFPO further integrates conditional risk control and consistency constraints along value flow trajectories. Experiments on dialogue, math reasoning, and scientific tasks show that DFPO outperforms PPO, FlowRL, and other robust baselines under noisy supervision, achieving improved training stability and generalization.

Problem

• distributional reinforcement learning
• out-of-domain generalization
• noisy supervision
• value modeling
• LLM post-training

Innovation

• Distributional Reinforcement Learning
• Value Flow Field
• Conditional Risk Control
• Consistency Constraints
• LLM Post-Training