Long Context In-Context Compression by Getting to the Gist of Gisting

2025-04-11

AI Summary

Existing gisting methods, while requiring no modification to decoder-only Transformer architectures, suffer substantial performance degradation in long-context compression—primarily due to information-flow disruption and attention misallocation. This paper introduces GistPool: the first method to explicitly identify and address these mechanisms. GistPool integrates learnable subset selection, weighted pooling, and semantic refinement to achieve lightweight context compression without any architectural changes. Grounded in attention analysis and information-flow modeling, it remains fully compatible with standard inference frameworks. Experiments across long-context tasks—including document question answering and code reasoning—demonstrate that GistPool significantly outperforms prior gisting approaches. At a 3× compression ratio, it retains 98% of the original model’s performance and matches or exceeds the accuracy of average pooling—a strong baseline—while introducing negligible computational overhead.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Data Compression
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-context compression method with no architectural modification to the decoder transformer, is a promising approach due to its simplicity and compatibility with existing frameworks. While effective for short instructions, we demonstrate that gisting struggles with longer contexts, with significant performance drops even at minimal compression rates. Surprisingly, a simple average pooling baseline consistently outperforms gisting. We analyze the limitations of gisting, including information flow interruptions, capacity limitations and the inability to restrict its attention to subsets of the context. Motivated by theoretical insights into the performance gap between gisting and average pooling, and supported by extensive experimentation, we propose GistPool, a new in-context compression method. GistPool preserves the simplicity of gisting, while significantly boosting its performance on long context compression tasks.

Problem

• Gisting struggles with long context compression performance
• Existing methods add complexity hindering practical LLM adoption
• Average pooling outperforms gisting in compression tasks

Innovation

• GistPool enhances long context compression
• Average pooling outperforms gisting baseline
• No architectural changes to decoder transformer