PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression

2025-04-23

AI Summary

To address the high inference cost of large language models (LLMs) induced by excessively long prompts, this paper proposes PIS, a dynamic prompt compression framework grounded in attention mechanisms. PIS innovatively integrates native LLM attention scores with importance sampling to establish a two-tier compression architecture: at the token level, a lightweight 9-layer reinforcement learning network drives adaptive token sampling; at the sentence level, Russian Roulette sampling preserves semantic integrity while optimizing structural coherence. By systematically modeling token importance, PIS avoids the pitfalls of heuristic truncation and abstractive summarization. Evaluated across diverse benchmarks, PIS achieves state-of-the-art compression performance—substantially reducing context length—while unexpectedly improving both inference efficiency and generation quality. These results empirically validate that structured prompt compression yields positive gains for LLM inference.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have achieved remarkable progress, demonstrating unprecedented capabilities across various natural language processing tasks. However, the high costs associated with such exceptional performance limit the widespread adoption of LLMs, highlighting the need for prompt compression. Existing prompt compression methods primarily rely on heuristic truncation or abstractive summarization techniques, which fundamentally overlook the intrinsic mechanisms of LLMs and lack a systematic evaluation of token importance for generation. In this work, we introduce Prompt Importance Sampling (PIS), a novel compression framework that dynamically compresses prompts by sampling important tokens based on the analysis of attention scores of hidden states. PIS employs a dual-level compression mechanism: 1) at the token level, we quantify saliency using LLM-native attention scores and implement adaptive compression through a lightweight 9-layer reinforcement learning (RL) network; 2) at the semantic level, we propose a Russian roulette sampling strategy for sentence-level importance sampling. Comprehensive evaluations across multiple domain benchmarks demonstrate that our method achieves state-of-the-art compression performance. Notably, our framework serendipitously enhances reasoning efficiency through optimized context structuring. This work advances prompt engineering by offering both theoretical grounding and practical efficiency in context management for LLMs.

Problem

• Efficient prompt compression for large language models
• Dynamic token importance sampling using attention scores
• Dual-level compression for optimized context structuring

Innovation

• Dynamic prompt compression using attention scores
• Dual-level token and semantic compression mechanism
• Lightweight RL network for adaptive compression