FastE: Readout-Triggered Token Compression for LLM Embedding Inference

2026-09-08

AI Summary

研究解决了LLM嵌入模型中的前缀冗余问题，通过FastE方法压缩令牌，减少计算成本同时保持性能。

Abstract

In this study, we identify depth-dependent prefix redundancy in final-readout LLM embedding models, notably across representative backbones including Qwen3-Embedding and Qwen3-VL-Embedding. We find that removing prefix states is substantially more damaging in shallow layers than at greater depth, showing that prefix states become increasingly compressible as the prefix and readout states propagate through the network. To this end, we introduce FastE, a training-free, plug-and-play method. FastE uses a shared fixed threshold on batch-mean readout-prefix alignment as a lightweight online heuristic for selecting when compression occurs, and ranks prefix states by the attention scores they receive from the readout position to determine which states are retained in subsequent layers. Our evaluations demonstrate FastE's ability to substantially reduce computational costs: on NarrativeQA with Qwen3-Embedding-0.6B, it reduces decoder-backbone FLOPs by 40.11% while retaining 99.53% of Full Forward nDCG@10. Across five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks, the quality-efficiency trade-off is directly customizable through the maximum removal ratio without retraining. We believe FastE offers practical value for scalable embedding generation in retrieval, indexing, clustering, and multimodal representation systems.

Problem

• depth-dependent prefix redundancy
• LLM embedding models
• computational costs

Innovation

• Readout-Triggered Token Compression
• Batch-Mean Readout-Prefix Alignment
• Attention Scores
• Maximum Removal Ratio
• Scalable Embedding Generation