Stellar: Scalable Multimodal Document Retrieval for Natural Language Queries

2026-06-18

AI Summary

This work addresses the high memory overhead and poor scalability of multi-vector representations in multimodal document retrieval. To overcome these limitations, the authors propose Stellar, a novel framework that leverages high-quality lexical representations for efficient candidate pre-filtering and integrates a balanced clustering–based disk embedding storage layout with a dynamic loading strategy to enable late interaction retrieval. The approach achieves state-of-the-art retrieval effectiveness while reducing memory consumption and query latency by one to two orders of magnitude across four established benchmarks and a newly introduced large-scale dataset, substantially enhancing deployability in real-world systems.

Abstract

Multimodal document retrieval--selecting the most relevant multimodal document from a large corpus to answer a natural language query--plays an essential role in Retrieval-Augmented Generation (RAG) systems. State-of-the-art methods represent each document and query with multiple token-level embeddings and use late interaction to achieve high effectiveness. However, such multi-vector representations incur substantial memory overhead during retrieval, leading to poor scalability and hindering real-world deployment. In this paper, we present Stellar, a scalable multimodal document retrieval framework that stores token-level document embeddings on disk and loads only a small set of candidate embeddings into memory for late interaction. Stellar comprises two key components: (i) Lexical Representation-based Filtering (LRF), which fine-tunes a Multimodal Large Language Model (MLLM) as a sparse encoder to produce high-quality lexical representations, enabling efficient and effective document filtering to significantly reduce the candidate set; (ii) Efficient Disk-backed Late Interaction (DLI), which designs an on-disk token embedding storage layout guided by a balanced clustering algorithm, and dynamically loads only the necessary token embeddings into memory using a simple yet effective cost model. Extensive experiments on four real-world benchmarks and a newly presented large-scale dataset demonstrate that Stellar reduces memory overhead and query latency by 1-2 orders of magnitude compared to existing methods without compromising retrieval effectiveness.

Problem

• multimodal document retrieval
• memory overhead
• scalability
• late interaction
• retrieval-augmented generation

Innovation

• multimodal retrieval
• disk-backed embeddings
• late interaction
• lexical representation
• scalable RAG