From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG

2026-05-14

AI Summary

This work addresses the limitations of existing visual Retrieval-Augmented Generation (RAG) systems, which support only image- or scene-level retrieval and struggle with fine-grained queries while offering no traceability for errors. To overcome these challenges, we propose GranuRAG, the first framework that treats visual elements as first-class retrieval units. GranuRAG employs a three-stage mechanism—comprising element detection, multi-granularity cross-modal alignment, and attribution-constrained generation—to enable verifiable multimodal question answering. We introduce GranuVistaVQA, a new benchmark dataset featuring element-level annotations, and demonstrate that GranuRAG outperforms six strong baselines by up to 29.2% on this benchmark, substantially improving both fine-grained response accuracy and system interpretability.

Abstract

Multimodal Retrieval-Augmented Generation (RAG) systems retrieve evidence at coarse granularities (entire images or scenes), creating a mismatch with fine-grained user queries and making failures unverifiable. We introduce GranuVistaVQA, a multimodal benchmark featuring real-world landmarks with element-level annotations across multiple viewpoints, capturing the partial observation challenge where individual images contain only subsets of entities. We further propose GranuRAG, a multi-granularity framework that treats visual elements as first-class retrieval units through three stages: element-level detection and classification, multi-granularity cross-modal alignment for evidence retrieval, and attribution-constrained generation. By grounding retrieval at the element level rather than relying on implicit attention, our approach enables transparent error diagnosis. Experiments demonstrate that GranuRAG achieves up to 29.2% improvement over six strong baselines for this task.

Problem

• Multimodal RAG
• evidence retrieval
• fine-grained query
• verifiability
• granularity mismatch

Innovation

• multi-granularity retrieval
• element-level grounding
• multimodal RAG
• attribution-constrained generation
• verifiable reasoning