Prompt Compression via Activation Aggregation

2026-07-09

AI Summary

This work proposes a task-specific prompt compression method that reduces computational overhead during large language model inference without reprocessing the original prompt sequence. The approach extracts activation vectors from intermediate layers and generates a single compressed vector through learnable weighted aggregation, which is then injected into an early layer of the model to replace the original prompt. The study demonstrates that activations from intermediate layers can be effectively transferred to shallower layers and that their weighted combination robustly encodes recoverable semantic information. Experimental results show that this method substantially decreases per-query computation for fixed instructional prompts while incurring less than a 2% drop in task accuracy.

Abstract

Large language models process prompts by propagating activations through dozens of layers before generating a response. We ask whether the task-relevant information contained in an instruction prompt can be compressed into a single activation vector and re-injected into the model, replacing the original token sequence? We show this is achievable using a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer of the target LLM. The compressed vector preserves task-relevant information, incurring an accuracy drop of under $2\%$ relative to full prompt processing. Beyond its practical implications, including reducing per-query computation for fixed instruction prompts without reprocessing the original token sequence, our analysis reveals structure in the activation space of LLMs: (i) mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; (ii) a single activation vector encodes a quantifiable and recoverable amount of semantic information; (iii) a weighted sum of activations is a robust representation compressor.

Problem

• Prompt Compression
• Activation Aggregation
• Large Language Models
• Semantic Information
• Activation Vector

Innovation

• prompt compression
• activation aggregation
• large language models
• representation transfer
• semantic encoding