Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning

2026-06-30

AI Summary

This work addresses a critical limitation in existing post-training methods for medical multimodal large language models, which overly prioritize final answer correctness while neglecting the optimization of intermediate reasoning steps, thereby suffering from cascading failures triggered by early errors. To mitigate this, we propose Medical Reasoning-aware Policy Optimization (MRPO), the first reinforcement learning framework tailored for medical multimodal reasoning that incorporates step-aware feedback. MRPO introduces step-level process rewards, applying exponentially stronger penalties to ineffective early reasoning even when the final answer is incorrect, thus effectively interrupting error propagation. Experimental results demonstrate that MRPO consistently outperforms standard GRPO and state-of-the-art reinforcement learning baselines across three multimodal LLM backbones. Notably, on Qwen3-VL-8B-Instruct, it surpasses HuatuoGPT-Vision-34B by 2.79 points and reduces early reasoning failure rates from 64.0% to 13.0%.

Abstract

Recent multimodal large language models have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantly outcome-centric, relying on final answer correctness or sequence-level preferences. This suffers from sparse credit assignment, making it difficult to optimize the reasoning process essential for clinical applications. Our analysis reveals that cascading errors from early-stage reasoning failures are a leading cause of incorrect predictions in medical visual question answering (VQA) benchmarks. Motivated by this, we propose Medical Reasoning-aware Policy Optimization (MRPO), an RL algorithm that incorporates step-wise process rewards. When the final answer is incorrect, MRPO assigns exponentially larger penalties to tokens in earlier invalid reasoning steps, breaking failure cascades without compromising successful paths. Across three multimodal LLM backbones, MRPO consistently outperforms standard GRPO and a recent RL baseline, and on Qwen3-VL-8B-Instruct even surpasses substantially larger medical MLLMs such as HuatuoGPT-Vision-34B by 2.79 points. Moreover, MRPO reduces early-stage reasoning failures from 64.0% to 13.0%, showing that targeted mitigation of cascading failures improves both reasoning quality and final answer accuracy. Our code is available at https://github.com/dmis-lab/MRPO

Problem

• failure cascades
• medical multimodal reasoning
• credit assignment
• visual question answering
• reasoning process

Innovation

• step-aware reinforcement learning
• failure cascades
• medical multimodal reasoning
• process rewards
• reasoning error mitigation