FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

2025-10-01

AI Summary

Existing neural audio codecs suffer severe semantic information loss at ultra-low frame rates (<12.5 Hz), limiting their integration into speech-language models. To address this, we propose a semantic-aware dynamic frame-rate codec framework. Our method features an ASR-guided dual-stream encoder that disentangles acoustic and semantic representations; incorporates a Transformer-based bottleneck module to enhance high-level semantic modeling; and employs semantic similarity–driven adaptive frame merging to support continuously adjustable frame rates from 3 to 12.5 Hz. Experiments demonstrate consistent and significant improvements over baselines across average frame rates of 6.25, 8.3, and 12.5 Hz, achieving simultaneous gains in both speech reconstruction quality and semantic fidelity. Furthermore, the codec is successfully integrated into a language-model-driven text-to-speech system, validating its practical applicability in end-to-end spoken language processing pipelines.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Neural audio codecs are foundational to speech language models. It is expected to have a low frame rate and decoupled semantic and acoustic information. A lower frame rate codec can reduce the computational cost of speech language models by shortening the sequence length. Recent studies have developed 12.5Hz low-frame-rate audio codecs, but even lower frame rate codecs remain underexplored. We find that a major challenge for very low frame rate tokens is missing semantic information. This paper introduces FlexiCodec to address this limitation. FlexiCodec improves semantic preservation with a dynamic frame rate approach and introduces a novel architecture featuring an ASR feature-assisted dual stream encoding and Transformer bottlenecks. With dynamic frame rates, it uses less frames at information-sparse regions through adaptively merging semantically similar frames. A dynamic frame rate also allows FlexiCodec to support inference-time controllable frame rates between 3Hz and 12.5Hz. Experiments on 6.25Hz, 8.3Hz and 12.5Hz average frame rates confirm that FlexiCodec excels over baseline systems in semantic information preservation and delivers a high audio reconstruction quality. We also validate the effectiveness of FlexiCodec in language model-based TTS. Demos are available at: https://flexicodec.github.io

Problem

• Addresses semantic loss in ultra-low frame rate neural audio codecs
• Introduces dynamic frame rate approach for adaptive information compression
• Enables controllable frame rates while maintaining audio reconstruction quality

Innovation

• Dynamic frame rate adaptively merges similar frames
• ASR feature-assisted dual stream encoding architecture
• Transformer bottlenecks enhance semantic and acoustic processing