Efficient Long Context Language Model Retrieval with Compression

2024-12-24

AI Summary

To address the inefficiency and high computational cost of long-context language models (LCLMs) in retrieval tasks caused by excessively long inputs, this paper proposes CoLoR—a fully differentiable, end-to-end paragraph compression method. Its core innovation lies in the first joint modeling of preference optimization and explicit length regularization, enabling the compressor to dynamically prune redundant context while preserving semantic integrity. CoLoR is trained on synthetically generated discriminative data to align compressed passages with human-preferred relevance signals. Evaluated on nine standard retrieval benchmarks, CoLoR achieves an average 6% improvement in retrieval accuracy while reducing input length by a factor of 1.91. This yields significant inference acceleration and lower computational overhead, thereby achieving synergistic optimization of retrieval effectiveness and context efficiency.

Technology Category

• Natural Language Processing: Summarization
• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Long Context Language Models (LCLMs) have emerged as a new paradigm to perform Information Retrieval (IR), which enables the direct ingestion and retrieval of information by processing an entire corpus in their single context, showcasing the potential to surpass traditional sparse and dense retrieval methods. However, processing a large number of passages within in-context for retrieval is computationally expensive, and handling their representations during inference further exacerbates the processing time; thus, we aim to make LCLM retrieval more efficient and potentially more effective with passage compression. Specifically, we propose a new compression approach tailored for LCLM retrieval, which is trained to maximize the retrieval performance while minimizing the length of the compressed passages. To accomplish this, we generate the synthetic data, where compressed passages are automatically created and labeled as chosen or rejected according to their retrieval success for a given query, and we train the proposed Compression model for Long context Retrieval (CoLoR) with this data via preference optimization while adding the length regularization loss on top of it to enforce brevity. Through extensive experiments on 9 datasets, we show that CoLoR improves the retrieval performance by 6% while compressing the in-context size by a factor of 1.91.

Problem

• Long Context Language Models
• Information Retrieval Efficiency
• Computational Resource Reduction

Innovation

• Long Context Language Models
• Compression Method
• Information Retrieval Efficiency