Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

2026-06-15

AI Summary

This work investigates how the position of retrieved passages influences answer generation in multimodal retrieval-augmented question answering, particularly whether a “middle information loss” phenomenon—akin to that observed in purely textual settings—exists. The authors propose the first controllable golden-position probing protocol to systematically evaluate the positional sensitivity of vision-language models on knowledge-based visual question answering (KB-VQA). Experiments reveal a pronounced primacy bias rather than a U-shaped performance curve, uncovering a “Lost at the End” effect: performance drops significantly for passages placed at the end, stemming from the instruction-tuned reader’s strong reliance on prompt slot 0. Across three open-source 7B/8B models and two benchmarks, placing relevant passages first yields gains of 16–26 points over last-position placement. Retrieval-side re-ranking fails to mitigate this issue, indicating the necessity of reader-side interventions. Code and the probing protocol are publicly released.

Abstract

Knowledge-based visual question answering (KB-VQA) lets vision-language systems answer questions that exceed their parametric knowledge by conditioning a reader on passages retrieved from a Wikipedia-scale knowledge base. In pure-text long-context LLMs, retrieved-context use follows the U-shaped "lost-in-the-middle" effect of Liu et al. (2024): information at the start and end of context is used, the middle is lost. Whether this transfers to deployed multimodal KB-VQA is open. To close this gap, we design the first controlled probe of reader-side position dependence in multimodal KB-VQA: a gold-position protocol in which only the gold passage's prompt slot varies within question. We run it on three open-source 7B/8B VLM readers and two KB-VQA benchmarks at k up to 20. The shape flips from U to primacy: gold-at-first beats gold-at-last by 16 to 26 points on every reader-by-benchmark cell, an effect we call "Lost at the End". Three targeted ablations narrow the cause: a text-only control shows the multimodal setting amplifies an already-present text-mode primacy 2.2 to 4.5 times, and image-position and distractor-shuffle ablations together pin the locus to prompt slot 0 of the instruction-tuned reader. On a frozen reader, three retrieval-side fixes (MMR, oracle reranking, rank-based reordering) all leave the gap intact (no separable improvement). Our findings indicate that recall@k is the wrong metric for deployed KB-VQA and that closing the gap requires reader-side intervention; we release our protocol as a controlled instrument for evaluating such interventions.

Problem

• multimodal retrieval
• position bias
• visual question answering
• retrieval-augmented generation
• primacy effect

Innovation

• primacy bias
• multimodal retrieval
• KB-VQA
• position dependence
• reader-side intervention