AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrieval

2026-08-09

AI Summary

This work addresses the high storage and computational costs in multi-vector visual document retrieval, where hundreds of image patch embeddings per page lead to inefficiency, and existing training-free compression methods suffer severe performance degradation under high compression ratios or fail to prioritize critical regions. The authors propose AnchorFold, a training-free “focus-and-fold” framework that uniquely integrates recursive attention propagation with cross-head and cross-layer centrality analysis. By leveraging self-attention graphs to identify high-centrality anchor tokens and aggregating non-anchor tokens via centrality-weighted clustering to their nearest anchors, AnchorFold achieves efficient compression while preserving structurally salient information. Evaluated on ViDoRe v1/v2 and REAL-MM-RAG, the method retains 98.3% of NDCG@5 at 5× compression and 92.4% at 20×, substantially outperforming current training-free baselines.

Abstract

Multi-vector vision-language retrievers enable fine-grained Visual Document Retrieval (VDR) through late interaction, but storing and scoring hundreds of visual patch embeddings per page incurs substantial overhead. Existing training-free methods rely on pruning or merging: pruning degrades sharply under aggressive compression, whereas merging does not explicitly prioritize important regions when forming representatives. We introduce AnchorFold, a training-free focus-then-fold framework for document-side index compression. AnchorFold applies Recursive Attention Propagation over visual self-attention graphs, performing multi-step propagation within each attention head and integrating scores across heads and layers. The focus stage selects the highest-centrality tokens as anchors. The fold stage assigns remaining tokens to their most similar anchors in the normalized retrieval space and summarizes each anchor-centered group through centrality-weighted aggregation. This preserves non-anchor contributions while concentrating capacity on structurally important tokens. Across ViDoRe v1/v2 and REAL-MM-RAG with three diverse retrieval backbones, AnchorFold consistently outperforms all evaluated training-free baselines at $γ\leq 0.20$. On ViDoRe v1/v2, it retains 98.3% of full-index NDCG@5 on average at $5\times$ compression, achieving near-lossless compression, and 92.4% at $20\times$ compression.

Problem

• Visual Document Retrieval
• multi-vector retrieval
• index compression
• late interaction
• embedding pruning

Innovation

• AnchorFold
• Recursive Attention Propagation
• multi-vector retrieval
• index compression
• visual document retrieval