Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning

2026-04-28

AI Summary

This work addresses the tendency of vision-language models to over-rely on linguistic priors during instruction following, often generating hallucinated content unsupported by ambiguous visual inputs. To mitigate this issue, the authors propose a dual-stream decoding framework that concurrently models instruction-driven and evidence-driven token distributions during generation. They introduce, for the first time, a contrastive gating mechanism based on symmetric KL divergence to dynamically fuse these two streams, effectively suppressing outputs lacking visual grounding while preserving coherent generations where both streams agree. The method significantly enhances visual grounding in generated responses, consistently outperforming existing decoding strategies across multiple benchmarks—including POPE, MME, and VQAv2—by substantially reducing hallucination rates while improving task accuracy and reasoning performance.

Abstract

Vision-Language Models (VLMs) exhibit strong performance in instruction following and open-ended vision-language reasoning, yet they frequently generate fluent outputs that are weakly grounded in visual evidence. Prior works have shown that instruction prompting further worsens this issue by amplifying language priors, especially when the visual signal is uncertain or ambiguous. To address this challenge, we propose a decoding framework that explicitly balances linguistic informativeness and visual faithfulness during generation. Our method, Instruction-Evidence Contrastive Dual-Stream Decoding (IECD2), maintains two parallel probability distributions of tokens at each decoding step: an instruction-driven stream that promotes expressive and informative responses, and an evidence-driven stream that enforces strict grounding in the image. These two streams are adaptively fused using a symmetric KL-based contrast-based gate, which suppresses tokens favored by language priors but unsupported by visual evidence, while preserving them when both distributions agree. We evaluate IECD2 on multiple datasets spanning various generative vision-language reasoning tasks such as captioning and visual question answering, including POPE, MME, VQAv2, AMBER, MS-COCO, and LLaVA-Bench. IECD2 demonstrates consistent improvements in task accuracy and reasoning performance, alongside a substantial reduction in hallucination across all evaluation metrics compared to state-of-the-art decoding approaches.

Problem

• vision-language grounding
• hallucination
• language priors
• visual evidence
• instruction following

Innovation

• dual-stream decoding
• visual grounding
• hallucination mitigation
• contrastive fusion
• vision-language reasoning