Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning

2026-02-22

AI Summary

This work addresses the challenges of entropy collapse and policy degradation in multimodal large language models when applied to reinforcement learning, which stem from vast state spaces and sparse rewards. To mitigate these issues, the authors propose CalibRL, a framework that enables controlled exploration through a hybrid policy design. CalibRL incorporates a distribution-aware advantage-weighting mechanism to preserve exploratory diversity and introduces an asymmetric activation function—such as LeakyReLU—informed by expert knowledge to steer policy updates. This approach effectively reduces distributional mismatch between the learned policy and expert trajectories while maintaining policy entropy and balancing exploration with exploitation. Evaluated across eight in-domain and out-of-domain benchmarks, CalibRL consistently outperforms existing methods, demonstrating both strong effectiveness and generalization capability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads to entropy collapse, policy degradation, or over-exploitation of suboptimal behaviors. This necessitates an exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. In this paper, we propose CalibRL, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, a distribution-aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Meanwhile, the asymmetric activation function (LeakyReLU) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy in a guided manner and clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding convergence to erroneous patterns. Importantly, these designs help alleviate the distributional mismatch between the model's policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation. Extensive experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements, validating the effectiveness of our controllable hybrid-policy RLVR training. Code is available at https://github.com/zhh6425/CalibRL.

Problem

• Reinforcement Learning
• Multi-Modal Reasoning
• Exploration
• Entropy Collapse
• Sparse Rewards

Innovation

• hybrid-policy RLVR
• controllable exploration
• distribution-aware advantage weighting
• asymmetric activation
• policy entropy calibration