🤖 AI Summary
To address the high computational and parameter overhead of mainstream neural audio codecs (NACs) and the insufficient performance of existing lightweight streaming approaches, this paper proposes SpecTokenizer—the first lightweight streaming NAC that enables multi-scale modeling directly in the compressed spectral domain. Its novel CNN-RNN alternating architecture jointly performs feature extraction and temporal modeling on low-dimensional spectral representations, significantly reducing computational redundancy. At 4 kbps, SpecTokenizer matches or surpasses the performance of state-of-the-art lightweight NACs while requiring only 20% of their computational cost and 10% of their parameters. Under identical resource constraints, it achieves substantial improvements in both PSNR and MOS scores. This work establishes a new paradigm for efficient audio compression and representation on edge devices.
📝 Abstract
Neural Audio Codecs (NACs) have gained growing attention in recent years as technologies for audio compression and audio representation in speech language models. While mainstream NACs typically require G-level computation and M-level parameters, the performance of lightweight and streaming NACs remains underexplored. This paper proposes SpecTokenizer, a lightweight streaming codec that operates in the compressed spectral domain. Composed solely of alternating CNN and RNN layers, SpecTokenizer achieves greater efficiency and better representational capability through multi-scale modeling in the compressed spectrum domain. At 4 kbps, the proposed SpecTokenizer achieves comparable or superior performance compared to the codec with state-of-the-art lightweight architecture while requiring only 20% of the computation and 10% of the parameters. Furthermore, it significantly outperforms the codec when using similar computational and storage resources.