Probing neural audio codecs for distinctions among English nuclear tunes

2026-03-14

AI Summary

This study investigates whether neural audio codecs can effectively encode the pitch contours of English sentence-final nuclear intonation. Leveraging the intonation-annotated corpus from Cole et al. (2023), we evaluate the discriminative capacity of both linear and nonlinear probes applied to the models’ latent representations and hierarchical vector quantization codebooks across five robust intonation clusters. Results show that linear probes achieve an average accuracy of 0.45 in the five-way classification task and 0.74–0.89 in binary rising-versus-falling intonation classification. Intonation information is distributed across all codebook levels, challenging the assumed dichotomy between “semantic” and “acoustic” codebooks. This work provides the first systematic assessment of neural codecs’ capabilities and limitations in fine-grained intonational modeling, revealing a significant performance gap compared to human-level perception.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

State-of-the-art spoken dialogue models (Défossez et al. 2024; Schalkwyk et al. 2025) use neural audio codecs to "tokenize" audio signals into a lower-frequency stream of vectorial latent representations, each quantized using a hierarchy of vector codebooks. A transformer layer allows these representations to reflect some time- and context-dependent patterns. We train probes on labeled audio data from Cole et al. (2023) to test whether the pitch trajectories that characterize English phrase-final (nuclear) intonational tunes are among these patterns. Results: Linear probes trained on the unquantized latents or some of the associated codewords yield above-chance accuracy in distinguishing eight phonologically specified nuclear tunes with monotonal pitch accents (top average test accuracy (TATA): 0.31) and the five clusters of these tunes that are robust in human speech production and perception (TATA: 0.45). Greater accuracy (TATAs: 0.74-0.89) is attained for binary distinctions between classes of rising vs. falling tunes, respectively used for questions and assertions. Information about tunes is spread among all codebooks, which calls into question a distinction between 'semantic' and 'acoustic' codebooks found in the literature. Accuracies improve with nonlinear probes, but discrimination among the five clusters remains far from human performance, suggesting a fundamental limitation of current codecs.

Problem

• neural audio codecs
• nuclear tunes
• intonational distinctions
• pitch trajectories
• probing

Innovation

• neural audio codecs
• probing
• nuclear intonation
• vector quantization
• pitch trajectories