Problem
Research questions and friction points this paper is trying to address.
multimodal retrieval
multi-vector representation
representation capacity
sample-adaptive
AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval
🤖 AI Summary
本文提出Sample-Adaptive Multi-Vector Representation (SAMVR)来解决多模态检索中不同样本需要不同表示容量的问题，通过AdaptiveEmbed框架和Utility Policy Optimization方法实现。
📝 Abstract
Multi-vector representations have emerged as an effective paradigm for multimodal retrieval, representing each sample with multiple complementary embeddings to capture fine-grained cross-modal information. However, existing approaches typically employ a fixed representation capacity, assigning the same number of vectors to all samples regardless of their individual retrieval demands. Such a fixed-capacity formulation overlooks the fact that different samples may require different amounts of representation capacity for effective retrieval. In this work, we introduce \emph{Sample-Adaptive Multi-Vector Representation} (SAMVR), a new problem setting for multimodal retrieval that studies how multi-vector representation capacity can be allocated at the sample level. Under SAMVR, each sample is represented by a \emph{content-adaptive embedding set} (CAES), whose capacity is determined according to the sample-specific retrieval utility of additional representation vectors. To instantiate SAMVR, we propose \emph{AdaptiveEmbed}, a unified framework for learning sample-adaptive multi-vector representations. AdaptiveEmbed learns structured multi-vector representations through \emph{Multi-Group Contrastive Learning} (MGCL) with the symmetric \emph{set-to-set similarity} (SetSim), and further employs \emph{Utility Policy Optimization} (UPO) to determine sample-specific representation capacity via \emph{Marginal Utility Allocation} (MUA). Experiments across multimodal retrieval benchmarks involving image, text, video, and audio show that sample-adaptive capacity allocation achieves overall better retrieval performance than fixed-capacity multi-vector representations, validating the effectiveness of SAMVR for multimodal retrieval. These results establish SAMVR as a viable formulation for adaptive capacity allocation in multi-vector multimodal retrieval.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Sample-Adaptive Multi-Vector Representation
Multi-Group Contrastive Learning
Utility Policy Optimization
🔎 Similar Papers
No similar papers found.
