Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

2026-07-03

AI Summary

This work addresses the inefficiency in multimodal retrieval-augmented generation (RAG) systems, where premature invocation of vision-language models (VLMs) often leads to unnecessary computational cost or degraded accuracy. The authors propose a posterior selective modality upgrading mechanism that first generates an initial answer using only textual and tabular information, then employs a verifier to assess whether critical visual cues are missing. The VLM is activated only when the expected utility gain outweighs its computational cost. This approach challenges the assumption that modality relevance equates to modality utility and extends selective routing from retrieval and reasoning stages into the modality dimension itself, enabling finer-grained trade-offs between cost and accuracy. Experiments on MultiModalQA demonstrate that the method substantially reduces VLM invocations while recovering the accuracy of a full VLM pipeline and approaching the performance upper bound of an oracle upgrading strategy.

Abstract

Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities -- text, tables, and images. The dominant deployment choice is binary and made before the model has tried to answer: either run a cheap text(+table) pipeline, or pay for an expensive vision-language model (VLM) over every image. Recent adaptive systems improve on this by selecting the modality or fidelity pre-retrieval, from a question-conditioned predictor of which modality will be needed. We show that this is the wrong decision point. Through an oracle headroom analysis on MultiModalQA, we find that the relevance of a modality to a question is a weak predictor of whether that modality is actually needed to answer correctly: a large fraction of questions whose gold support includes an image are nonetheless answerable from text and tables alone, and a pre-retrieval router that escalates on apparent visual relevance over-escalates substantially relative to an oracle. We propose \textbf{post-hoc selective modality escalation}: answer cheaply from text and tables, run a verifier on the (query, draft answer, evidence) tuple that localizes which modality is missing, and pay for VLM evidence only there. A calibrated value-of-escalation router then decides whether the expected accuracy gain justifies the visual cost. On MultiModalQA, our router recovers the accuracy of an always-on VLM pipeline while issuing far fewer visual calls, and closes most of the gap to the oracle escalation rate. The result extends a routing-signal hierarchy established for retrieval depth and reasoning hops to a third axis -- modality -- under a single cost-aware selective-escalation view.

Problem

• Multimodal RAG
• Modality Utility
• Cost-Aware
• Selective Escalation
• Vision-Language Model

Innovation

• post-hoc selective escalation
• cost-aware multimodal RAG
• modality utility
• verification-based routing
• visual call reduction