EigenLI: Spectral Approximations to Late Interaction

2026-09-07

AI Summary

为解决ColBERT等模型的高索引成本和存储问题，EigenLI通过识别文档的主要特征方向并构建低维子空间来压缩表示，从而提高效率。

Abstract

Late-interaction models such as ColBERT achieve strong effectiveness by representing each document with many token-level vectors, but this expressivity leads to large indexing cost, storage footprints and expensive MaxSim scoring. We show that late-interaction representations exhibit an intrinsic low-rank structure: document token embeddings concentrate in a low-dimensional subspace that preserves most of the retrieval signal. Leveraging this observation, we introduce EigenLI, a spectral approximation framework that compresses late-interaction representations via document-specific low-dimensional subspaces. Unlike clustering or pooling methods, EigenLI identifies the dominant eigendirections of each document and uses them to construct reduced interaction representations. Empirically, $k$-EigenLI with $k \le 32$ outperforms k-means and Ward clustering based pooling methods on ColBERTv2 and AnswerAI-ColBERT-small; GTE-ModernColBERT exhibits a different tradeoff at $k=32$, where clustering methods perform better. The same spectral construction also yields EigenLI-SV, an ANN-compatible single-vector representation derived from the second-order summary of the reduced structure. Across multiple datasets and all three text models, EigenLI-SV consistently outperforms comparable single-vector surrogates such as MUVERA.

Problem

• late-interaction models
• indexing cost
• storage footprints
• MaxSim scoring
• low-rank structure

Innovation

• Spectral Approximation
• Late-interaction Models
• Low-dimensional Subspaces
• Eigendirections
• Single-vector Representation