Bottleneck Tokens for Unified Multimodal Retrieval

2026-04-13

AI Summary

This work addresses the insufficient semantic representation in decoder-only multimodal large language models for unified retrieval, which stems from implicit pooling and the absence of token-level compression guidance. To remedy this, the authors propose the Bottleneck Tokens (BToks) architecture, which employs a learnable explicit pooling mechanism and Condensation Mask attention to route all predictive signals through a set of bottleneck tokens. A generative information compression objective is introduced, transforming next-token prediction into dense, token-level supervision. Combined with contrastive fine-tuning, this approach substantially enhances semantic compression quality, achieving a score of 59.0 on the MMEB-V2 benchmark with a 2B-scale model—surpassing VLM2Vec-V2 by 3.6 points—and yielding gains up to 12.6 points on dense tasks such as video question answering.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).

Problem

• multimodal retrieval
• decoder-only MLLMs
• implicit pooling
• token-level compression
• structural gaps

Innovation

• Bottleneck Tokens
• Multimodal Retrieval
• Generative Information Condensation
• Explicit Pooling
• Token-level Supervision