Neural Audio Codec with Adjustable Token Temporal Resolution Using Sampling-Frequency-Independent Convolutional Layers

2026-07-02

AI Summary

This work addresses the inflexibility of conventional neural audio codecs, which require separate training for each token time resolution (TTR) and thus struggle to adapt to diverse modeling requirements. The paper introduces the first unified codec capable of supporting multiple TTRs within a single model. By employing sampling-rate-agnostic convolutional layers that dynamically generate kernels aligned with the target TTR—while keeping the quantizer fixed—the proposed approach enables seamless switching among various time resolutions without any retraining. This design overcomes the limitations of fixed-TTR architectures and demonstrates superior performance over a strong baseline that uses dedicated layers per TTR in environmental sound reconstruction tasks, thereby validating its efficacy and advantages.

Abstract

Discrete tokens obtained from neural audio codecs (NACs) have been used as compact representations in audio generation and understanding models. In such token-based systems, token temporal resolution (TTR), defined as the time interval between adjacent token frames, is important because it controls the trade-off between representing rapid acoustic events and reducing token-sequence length. However, most NACs are trained at a single TTR and require separate training for each TTR. This paper proposes a mechanism that enables a single NAC to operate at multiple TTRs using sampling-frequency-independent convolutional layers. The mechanism regards TTR as the sampling period of the token sequence and generates TTR-dependent convolutional kernels from a shared parameter set, while adjusting the kernel size and stride for each TTR. We incorporate the mechanism into Descript Audio Codec, leaving the quantizer unchanged. Experiments on environmental sound reconstruction show that the proposed model outperforms a single-model baseline that switches TTR-specific layers for each TTR.

Problem

• neural audio codec
• token temporal resolution
• sampling-frequency-independent
• multi-resolution
• convolutional layers

Innovation

• neural audio codec
• adjustable token temporal resolution
• sampling-frequency-independent convolution
• multi-resolution modeling
• shared-parameter kernel generation