Layer-wise Token Compression for Efficient Document Reranking

2026-05-19

AI Summary

This work addresses the high computational cost of Transformer-based cross-encoder reranking models when processing long query-document sequences, a challenge exacerbated by existing token compression methods that struggle to balance efficiency and effectiveness. The authors propose Layer-wise Token Compression (LTC), which introduces adaptive token pooling within intermediate layers of the cross-encoder for the first time. This approach not only enhances inference efficiency but also improves the model’s length invariance. LTC further functions as an implicit regularization mechanism and is successfully extended to listwise large language model reranking. Experiments on MS MARCO demonstrate that LTC increases queries per second (QPS) by 25% for passage reranking and by 116% for document reranking, while models trained with compression exhibit superior performance in cross-length transfer tasks.

Abstract

Transformer-based document cross-encoder rerankers are a central component of modern information retrieval systems. Despite their success, these models suffer from high computational costs due to processing long query-document sequences at inference time. A known approach to improve efficiency is token compression, which consists of aggregating groups of tokens together in the initial embedding layer, reducing the effective number of tokens, and making the computation faster. While token compression has proven to be successful for bi-encoder retrievers, we empirically observed that this approach may be ineffective for cross-encoder rerankers. In this paper, we propose Layer-wise Token Compression (LTC), which applies adaptive token pooling at intermediate transformer layers. Through extensive ablation studies on MS MARCO passage and document ranking tasks, we demonstrate that compression at middle layers preserves ranking quality while increasing inference QPS by up to 25% for passage ranking and up to 116% for document ranking. We also extend LTC to listwise LLM rerankers and show that the same approach can be easily applied to long-context listwise reranking, where the QPS improvements are even greater. More surprisingly, when applying rerankers trained on short passages to long-document ranking tasks, models trained with compression outperform their uncompressed counterparts, suggesting that compression may act as a beneficial regularizer that encourages length-invariant representations.

Problem

• document reranking
• token compression
• computational efficiency
• cross-encoder
• long-context ranking

Innovation

• Layer-wise Token Compression
• cross-encoder reranking
• token pooling
• length-invariant representation
• efficient inference