Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

2026-07-07

AI Summary

This work addresses the inefficiency of speech large language models during autoregressive decoding, which stems from the substantially longer length of speech sequences compared to text. To tackle this, the authors propose SpeechKV, the first method to directly apply a learnable pooling mechanism to compress key-value (KV) caches of speech tokens within the model’s internal layers, rather than performing early downsampling in adapter modules. This approach preserves fine-grained acoustic information while significantly accelerating inference, compressing speech representations to lengths comparable to text. The method achieves at least a 1.49× speedup in decoding—increasing with audio duration—and yields relative performance gains of 6.6% and 2.3% on out-of-domain entity recognition and OpenASR benchmarks, respectively, matching or slightly surpassing the uncompressed baseline in overall effectiveness.

Abstract

Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.

Problem

• Speech LLMs
• efficiency bottleneck
• autoregressive decoding
• sequence length
• KV cache

Innovation

• KV cache compression
• Speech LLMs
• learned pooling
• efficient decoding
• speech-text fusion