Dual-Flow Reinforcement Learning with State-Aware Exploration

2026-06-29

AI Summary

This work addresses the challenges posed by coexisting multimodal optimal actions and uncertain reward distributions in complex continuous control tasks, which often lead to value estimation bias and insufficient exploration. The authors propose a unified Actor-Critic framework that, for the first time, employs conditional flow matching to jointly model continuous reward distributions and multimodal policy distributions. Additionally, they introduce a state-aware exploration modulation mechanism based on policy entropy and the covariance of action uncertainty. Evaluated on the DeepMind Control Suite and Humanoid-Bench, the method significantly outperforms existing approaches based on diffusion models and flow models, achieving state-of-the-art performance.

Abstract

In complex continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions, making reliable value estimation and multimodal exploration challenging. Existing value estimation methods using unimodal Gaussians restrict expressiveness and yield biased estimates. Recent generative policies can represent multimodal actions but often collapse to a few modes and under-explore high-value areas of the action space. Motivated by these challenges, we propose Dual-Flow RL, a unified actor-critic framework that jointly models a continuous return distribution and a multimodal policy distribution using conditional flow matching (CFM). This design supports reliable value estimation and sustained multimodal exploration. To further enhance exploration, we introduce an Entropy-Covariance Exploration Regulator (ECER) that enables state-aware exploration regulation leveraging policy entropy and action-uncertainty covariance. Experiments on DeepMind Control Suite and Humanoid-Bench show that Dual-Flow RL achieves state-of-the-art performance on most tasks, significantly outperforming prior diffusion-based and flow-based methods.

Problem

• multimodal exploration
• value estimation
• continuous control
• return distribution
• reinforcement learning

Innovation

• Conditional Flow Matching
• Multimodal Policy
• State-Aware Exploration
• Return Distribution Modeling
• Entropy-Covariance Regulator