ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

2026-09-10

AI Summary

ZipCodec通过结合大规模WavLM蒸馏、改进的变压器架构等方法，解决了低帧率下的语音编码问题，实现了高质量的重建和下游任务性能。

Abstract

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

Problem

• neural audio codecs
• frame rate
• bitrate
• reconstruction quality

Innovation

• Ultra-Low-Frame-Rate
• Neural Speech Coding
• WavLM Distillation
• Latency-Aware Decoder