MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias

2026-06-16

AI Summary

This study addresses the tendency of multimodal large language models (MLLMs) to over-rely on textual inputs in image–text conflict scenarios, often disregarding visual evidence and thereby compromising visual grounding reliability. The work identifies a previously unreported phenomenon termed “late-stage textual override,” wherein models correctly encode visual information in intermediate layers but ultimately yield to textual dominance in later stages, producing erroneous outputs. To mitigate this issue, the authors propose Conflict-Aware Layer-Referenced Decoding (CALRD), a training-free inference-time method that dynamically detects shifts in prediction trajectories and restores visually grounded predictions suppressed by textual bias. Evaluated across five state-of-the-art MLLMs, CALRD improves accuracy by up to 9.4% on conflicting instances while preserving performance on standard benchmarks.

Abstract

When vision contradicts text, multimodal large language models (MLLMs) consistently favor text, even when images provide clear evidence otherwise. This bias poses risks for applications requiring visual grounding, yet its cause remains unclear. In this paper, we uncover a surprising finding: models often get it right initially, forming correct vision-based predictions in their intermediate layers, before changing their minds and favoring text in the final output. We call this "late-layer textual override". The visual information is encoded, it simply does not survive to the output. More intriguingly, we find that how predictions change reveals whether they're correct: 85% of failures shift toward text, while 89% of successes shift toward vision. This directional signature enables a simple but powerful intervention: when we detect a confident visual prediction being suppressed, we restore it. We propose CALRD (Conflict-Aware Layer Reference Decoding), a training-free method that recovers overridden predictions at inference time. Experiments across five MLLMs of varying architectures demonstrate up to 9.4% absolute improvements on conflict benchmarks while largely preserving standard performance, without training or external knowledge. It recovers what the model already knew but failed to preserve.

Problem

• multimodal large language models
• textual bias
• visual grounding
• late-layer override
• modality conflict

Innovation

• late-layer textual override
• visual grounding
• multimodal large language models
• inference-time intervention
• CALRD