CORA: Analyzing and bridging thinking-answer gap in Multimodal RLVR via Consistency-Oriented Reasoning Alignment

2026-06-12

AI Summary

This work addresses the semantic inconsistency between reasoning processes and final answers in verifiable reward-based multimodal reinforcement learning (RLVR), which undermines the reliability of generated reasoning trajectories. The study systematically identifies this “thought-answer misalignment” phenomenon for the first time and introduces a consistency-oriented reasoning alignment approach. By integrating a lightweight, plug-and-play Consistency Reward Model with a Hybrid Reward Advantage Separation (HRAS) strategy into the GRPO framework, the method jointly optimizes task performance and reasoning faithfulness. Experiments demonstrate that the proposed approach significantly improves accuracy across multiple multimodal reasoning benchmarks and mainstream large vision-language models (LVLMs), while effectively reducing inconsistency and yielding more trustworthy reasoning trajectories.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has successfully elicited the reasoning capabilities of large language models, motivating its extension to multimodal scenarios. Existing methods primarily focus on improving the visual coverage of reasoning traces and mitigating visual hallucinations, but underestimate the semantic inconsistency between the reasoning process and the final answer. In this paper, we delve into thinking-answer inconsistency in RLVR for large vision-language models (LVLMs), showing thorough analyses of rollouts collected throughout Group Relative Policy Optimization (GRPO) training process and post-RLVR evaluation outputs that this issue persists during training and remains present during inference. Motivated by the analysis, we propose Consistency-Oriented Reasoning Alignment (CORA), which introduces thinking-answer semantic consistency into RLVR through a lightweight plug-and-play consistency reward model, and further incorporates Hybrid Reward Advantage Splitting (HRAS) to stably coordinate task and consistency optimization. Extensive experiments across representative multimodal reasoning benchmarks and mainstream LVLMs show that CORA improves task performance while effectively mitigating thinking-answer inconsistency, leading to more faithful reasoning traces.

Problem

• thinking-answer inconsistency
• multimodal RLVR
• semantic consistency
• large vision-language models
• reasoning alignment

Innovation

• Consistency-Oriented Reasoning Alignment
• thinking-answer inconsistency
• multimodal RLVR
• Hybrid Reward Advantage Splitting
• large vision-language models