DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens

2025-02-17

AI Summary

Existing long-context processing methods for large language models (LLMs) suffer from high computational redundancy, while semantic compression approaches often ignore the dynamic variation of information density across contexts. Method: This paper proposes a context-aware soft token dynamic allocation mechanism that jointly leverages local perplexity and global attention weights to estimate block-level information richness, enabling adaptive allocation of learnable soft tokens; it further incorporates context importance-weighted aggregation to enhance preservation of critical semantics. Contribution/Results: Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines across multiple long-context benchmarks—achieving superior trade-offs between inference efficiency and semantic fidelity—thereby breaking the conventional paradigm of uniform context compression.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Data Compression

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) face computational inefficiencies and redundant processing when handling long context inputs, prompting a focus on compression techniques. While existing semantic vector-based compression methods achieve promising performance, these methods fail to account for the intrinsic information density variations between context chunks, instead allocating soft tokens uniformly across context chunks. This uniform distribution inevitably diminishes allocation to information-critical regions. To address this, we propose Dynamic Allocation of Soft Tokens (DAST), a simple yet effective method that leverages the LLM's intrinsic understanding of contextual relevance to guide compression. DAST combines perplexity-based local information with attention-driven global information to dynamically allocate soft tokens to the informative-rich chunks, enabling effective, context-aware compression. Experimental results across multiple benchmarks demonstrate that DAST surpasses state-of-the-art methods.

Problem

• Addresses computational inefficiencies in LLMs
• Improves context-aware compression techniques
• Dynamically allocates soft tokens effectively

Innovation

• Dynamic soft token allocation
• Context-aware compression technique
• Combines perplexity and attention