VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference

2026-08-09

AI Summary

This work addresses the substantial memory overhead of KV caching in long-context audio reasoning, where existing text-based compression methods often disrupt speech continuity or discard critical semantic information. To overcome these limitations, the authors propose VoxZip, the first training-free, two-stage semantic-anchored KV cache compression framework. VoxZip first leverages ASR transcripts as explicit semantic anchors to align and fuse audio tokens, then dynamically filters non-essential tokens via time-decayed cumulative attention. The method preserves over 90% of original performance even under 20× compression and achieves a 1.9× increase in inference throughput with a 3.3× reduction in peak memory usage at 4× compression. Extensive evaluations demonstrate that VoxZip consistently outperforms baseline approaches across both short- and long-form audio tasks.

Abstract

Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks. Despite this progress, their long-context inference remains severely bottlenecked by prohibitive KV cache memory demands. Existing text-centric compression methods struggle here, often disrupting speech continuity or discarding crucial semantic cues. To address this, we propose VoxZip, a train-free, two-stage semantic-anchored KV cache compression framework. The first stage uses automatic speech recognition (ASR) transcriptions as explicit semantic anchors to temporally align, compress, and fuse audio tokens, significantly reducing the initial KV cache while elevating token information density. To further improve the compression ratio, the second stage employs a dynamic filtering strategy based on temporally decayed accumulated attention to evict non-essential tokens while mitigating early-token bias. Comprehensive evaluations on Qwen3-Omni across six diverse audio benchmarks demonstrate the superiority of our approach. VoxZip excels in long-audio reasoning and consistently maintains high-fidelity perception on short-form tasks. Notably, it sustains over 90\% of the uncompressed baseline performance even under an aggressive 20x KV cache compression in long-context scenarios. Furthermore, at a 4x compression ratio, VoxZip yields a 1.9x increase in inference throughput alongside a 3.3x reduction in peak memory overhead. Code and models will be available at https://github.com/MM-Speech/VoxZip.

Problem

• KV cache compression
• long-context audio inference
• speech large language models
• semantic preservation
• memory bottleneck

Innovation

• KV cache compression
• semantic-anchored
• long-context audio
• train-free
• dynamic token filtering