Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings

2025-06-05

AI Summary

Patch-level embeddings in Vision-Document Retrieval (VDR) incur prohibitive memory overhead. Our systematic analysis reveals that query-agnostic token pruning fundamentally fails in VDR, whereas token merging holds greater promise. Method: We propose a lightweight, multi-dimensional token merging framework that jointly optimizes hierarchical clustering, cross-dimensional strategy search, and adapter-based fine-tuning—achieving Pareto-optimal trade-offs between memory efficiency and retrieval performance. Contribution/Results: Instantiated as Light-ColPali and Light-ColQwen2, our approach retains 98.2% of original retrieval effectiveness at just 11.8% memory cost, and sustains 94.6% effectiveness even at 2.8% memory—substantially outperforming all token-pruning baselines. This work is the first to uncover the intrinsic limitations of unsupervised token pruning in VDR and establishes token merging as a new paradigm for efficient embedding compression in VDR.

Technology Category

• Computer Vision: Image and Video Retrieval
• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Despite the strong performance of ColPali/ColQwen2 in Visualized Document Retrieval (VDR), it encodes each page into multiple patch-level embeddings and leads to excessive memory usage. This empirical study investigates methods to reduce patch embeddings per page at minimum performance degradation. We evaluate two token-reduction strategies: token pruning and token merging. Regarding token pruning, we surprisingly observe that a simple random strategy outperforms other sophisticated pruning methods, though still far from satisfactory. Further analysis reveals that pruning is inherently unsuitable for VDR as it requires removing certain page embeddings without query-specific information. Turning to token merging (more suitable for VDR), we search for the optimal combinations of merging strategy across three dimensions and develop Light-ColPali/ColQwen2. It maintains 98.2% of retrieval performance with only 11.8% of original memory usage, and preserves 94.6% effectiveness at 2.8% memory footprint. We expect our empirical findings and resulting Light-ColPali/ColQwen2 offer valuable insights and establish a competitive baseline for future research towards efficient VDR.

Problem

• Reducing excessive memory usage in Visual Document Retrieval (VDR)
• Evaluating token pruning and merging for embedding reduction
• Optimizing merging strategies to maintain retrieval performance

Innovation

• Random token pruning outperforms complex methods
• Token merging optimizes three-dimensional strategy combinations
• Light-ColPali/ColQwen2 reduces memory usage significantly