Beyond Single Embeddings: Capturing Diverse Targets with Multi-Query Retrieval

2025-11-04

AI Summary

Existing single-vector retrieval paradigms struggle to model multimodal relevance distributions—especially for ambiguous or polysemous queries—leading to limited coverage and accuracy. This work systematically identifies and characterizes this fundamental limitation for the first time. We propose AMER (Autoregressive Multi-Embedding Retrieval), a novel framework that autoregressively generates multiple complementary query embeddings to jointly capture complex, diverse query intents. Retrieval is performed end-to-end via similarity-weighted fusion of these embeddings. On synthetic multi-answer benchmarks, AMER achieves up to 4× higher performance than single-vector baselines; on real-world datasets, it yields up to a 21% relative improvement, with particularly pronounced gains when target documents exhibit high semantic divergence. This work establishes a new paradigm for multimodal intent modeling in neural retrieval.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Most text retrievers generate emph{one} query vector to retrieve relevant documents. Yet, the conditional distribution of relevant documents for the query may be multimodal, e.g., representing different interpretations of the query. We first quantify the limitations of existing retrievers. All retrievers we evaluate struggle more as the distance between target document embeddings grows. To address this limitation, we develop a new retriever architecture, emph{A}utoregressive emph{M}ulti-emph{E}mbedding emph{R}etriever (AMER). Our model autoregressively generates multiple query vectors, and all the predicted query vectors are used to retrieve documents from the corpus. We show that on the synthetic vectorized data, the proposed method could capture multiple target distributions perfectly, showing 4x better performance than single embedding model. We also fine-tune our model on real-world multi-answer retrieval datasets and evaluate in-domain. AMER presents 4 and 21% relative gains over single-embedding baselines on two datasets we evaluate on. Furthermore, we consistently observe larger gains on the subset of dataset where the embeddings of the target documents are less similar to each other. We demonstrate the potential of using a multi-query vector retriever and open up a new direction for future work.

Problem

• Addressing multimodal document distributions in text retrieval systems
• Overcoming limitations of single-vector query representations
• Improving retrieval for diverse relevant document targets

Innovation

• Autoregressively generates multiple query vectors
• Uses all predicted vectors for document retrieval
• Captures multimodal distributions of relevant documents