Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval

2026-07-05

AI Summary

This work addresses the high storage and computational costs in multi-vector vision-language retrieval caused by the large number of image-side tokens, while noting that existing compression methods often degrade critical object-level visual evidence. The authors propose SaMer, a novel framework that introduces, for the first time, an object-aware token merging mechanism. During training, object annotations serve as priors to cluster projected tokens into K representative centers; at inference, no object detector or ground-truth bounding boxes are required—only a lightweight fine-tuning of a shared projection layer preserves the original late-interaction interface. With K=64, SaMer reduces image tokens by over 93%, slashes ColPali’s storage overhead by 16.09×, and improves R@1 on both Flickr30K and MSCOCO, significantly outperforming existing compression baselines while also enhancing phrase-level localization capability.

Abstract

Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into $K$ representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With $K=64$, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by $16.09\times$, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.

Problem

• vision-language retrieval
• token compression
• object evidence preservation
• multi-vector retrieval
• visual tokens

Innovation

• token merging
• object-aware compression
• vision-language retrieval
• multi-vector retrieval
• late interaction