Generative Universal Multimodal Retrieval with Dual-role Identifiers

2026-08-13

AI Summary

Existing generative retrieval methods struggle in multimodal settings due to prefix errors, susceptibility to local optima, and insufficient accuracy, while also lacking a unified framework for handling text, images, and mixed modalities. To address these limitations, this work proposes DrIG, a novel framework that assigns each candidate item a residual quantized identifier functioning dually as both a sequence and a set. The sequence role explicitly models modality and semantic hierarchies through autoregressive decoding, while the set role provides prefix-agnostic relevance priors to mitigate local optima. By integrating unordered set reinterpretation, constrained beam search, and hybrid re-ranking strategies, DrIG achieves substantial improvements over current generative multimodal retrieval approaches on M-BEIR and text-image retrieval benchmarks, striking an effective balance between efficiency and effectiveness.

Abstract

Generative information retrieval (GIR) has emerged as a compelling alternative to the conventional index-retrieve-then-rank retrieval pipeline by training a generator to produce the identifiers of relevant items directly. Despite its promise, a number of open challenges still remain. First, constrained left-to-right decoding is vulnerable to prefix-level errors and local optima. Second, most prior GIR research remains largely unimodal, leaving instruction-aware retrieval across text, image, and mixed image-text items underexplored. Third, although discrete identifier-based GIR offers higher efficiency, its retrieval accuracy still lags behind that of the cutting-edge dense-vector-based retrieval methods. Motivated by these challenges, we propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers, which supports diverse retrieval tasks across multiple modalities and domains. Each candidate is assigned a single residual-quantized identifier that serves two complementary roles. In its sequential role, the identifier is decoded autoregressively, where the first token explicitly models modality and the remaining tokens capture progressively finer semantics. In its set-based role, the same tokens are reinterpreted as an unordered set to provide a prefix-independent relevance prior, which guides constrained beam search and alleviates local-optimum errors. Extensive experiments on the M-BEIR benchmark and the text-to-image evaluation datasets show that:(1)DrIG consistently outperforms state-of-the-art generative multimodal baselines across diverse tasks, while hybrid reranking achieves a favorable efficiency-effectiveness trade-off against strong dense retrievers. (2)Ablation and scaling analyses reveal how the base LMM, beam size, reranking depth, and fusion strategy affect retrieval performance, providing practical guidance for system design.

Problem

• generative information retrieval
• multimodal retrieval
• discrete identifiers
• instruction-aware retrieval
• prefix-level errors

Innovation

• Generative Information Retrieval
• Multimodal Retrieval
• Dual-role Identifiers
• Residual Quantization
• Constrained Beam Search