Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness

2026-04-20

AI Summary

This work addresses the inconsistency between reasoning processes and final answers in multimodal reinforcement learning by proposing a fine-grained reward mechanism based on trajectory supervision. The key innovation is the introduction of a Groupwise Ranking Reward, which, within a single forward pass, ranks all verified reasoning trajectories generated under the same prompt and dynamically redistributes rewards among them. This approach effectively balances stability, efficiency, and discriminative power. By integrating reinforcement learning with verifiable rewards (RLVR) and combining reward models (RMs) with generative rewards (GRs), the method significantly enhances reasoning reliability—improving conditional accuracy from 47.4% to 54.7% and outperforming existing RLVR and other trajectory supervision approaches.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves multimodal reasoning by rewarding verifiable final answers. Yet answer-correct trajectories may still rely on incomplete derivations, weak evidence, or statements that contradict their conclusions. This gap between answer correctness and reasoning validity, which we call reasoning-answer inconsistency, motivates trajectory supervision in multimodal RL. We compare two main approaches: reward models (RMs), and Generative Rewards (GRs). RMs are efficient and help early in training, but their gains weaken as the policy distribution shifts; GRs improve performance, but may give unstable rewards and computationally expensive. We therefore propose Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly. Groupwise comparison better separates stronger and weaker correct trajectories with lower judge overhead than GRs. Experiments show that RLVR aggravates reasoning-answer inconsistency, while trajectory supervision alleviates it. Groupwise Ranking Reward performs best overall, improving reliability-conditioned accuracy from 47.4% to 54.7% over RLVR.

Problem

• reasoning-answer inconsistency
• multimodal reasoning
• trajectory supervision
• reinforcement learning
• verifiable rewards

Innovation

• Groupwise Ranking Reward
• trajectory supervision
• reasoning-answer inconsistency
• multimodal reinforcement learning
• verifiable rewards