Multi-Branch Policy Optimization for Multimodal Large Language Models

2026-08-05

AI Summary

This work addresses the challenge that multimodal large language models in reinforcement learning suffer from degraded learning signals due to trajectory-level unified advantage estimation under high visual perceptual uncertainty. To mitigate this issue, the authors propose Multibranch Policy Optimization (MBPO), a novel framework that introduces a tree-structured reasoning mechanism into multimodal reinforcement learning for the first time. Specifically, MBPO constructs a reasoning tree at the vision–language decision boundary, where sibling branches explore alternative visual hypotheses and enable segment-level credit assignment through relative advantages across branches. Additionally, a temporal replay buffer is incorporated to enhance sample efficiency and training stability. Experimental results demonstrate that MBPO significantly outperforms existing baselines on multiple multimodal reasoning benchmarks and effectively alleviates credit assignment failure under conditions of high perceptual uncertainty.

Abstract

Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO.

Problem

• multimodal reasoning
• credit assignment
• perceptual uncertainty
• visual grounding
• reinforcement learning

Innovation

• Multi-Branch Policy Optimization
• reasoning trees
• segment-level credit assignment
• branch-relative advantages
• temporal replay buffer