Streaming Neural Speech Codecs through Time-Invariant Representations

2026-07-06

AI Summary

This work addresses the challenges faced by conventional neural speech codecs in low-latency scenarios, where heavy frame-level modeling burdens hinder the simultaneous optimization of reconstruction quality and computational efficiency. To overcome this, the authors propose TiCodec, a novel framework featuring a Time-Invariant Representation Extraction (TIRE) module that disentangles speech into time-varying and time-invariant components, substantially reducing frame-level modeling complexity while enabling streaming processing. By incorporating a Dual-TIRE multi-layer architecture that fuses complementary information from different encoder depths, along with factorized representation learning and a 660-ms chunk-based streaming inference strategy, the method significantly enhances both reconstruction fidelity and speaker similarity. Experimental results demonstrate that TiCodec achieves near non-streaming performance under streaming conditions, making it well-suited for low-latency speech generation systems.

Abstract

Neural speech codecs are increasingly used as intermediate representations in codec-based speech generation systems. TiCodec introduces a factorized representation that separates time-varying speech content from time-invariant information through a Time-Invariant Representation Extraction (TIRE) module, potentially reducing the amount of information that must be modeled at the frame-level. In this work, we investigate the nature of the information captured by TIRE representations and their suitability for low-latency speech processing. Using a series of probing tasks, we analyze the influence of the encoder layer and show that intermediate layers capture complementary speaker- and environment-related information while containing little linguistic content. We further study several segment selection strategies for TIRE training and demonstrate that cross-file sampling improves the robustness of invariant representations. Based on these findings, we propose Dual-TIRE, a multi-level architecture that exploits the complementarity of different encoder layers and improves speech reconstruction quality and speaker similarity. Finally, we evaluate TiCodec in a streaming inference setting using successive 660ms processing blocks. Results show that streaming operation can be achieved without significant degradation in reconstruction performance, highlighting the potential of factorized neural codec representations for future low-latency speech generation systems.

Problem

• neural speech codecs
• time-invariant representations
• low-latency speech processing
• streaming inference
• speech generation

Innovation

• Time-Invariant Representation
• Neural Speech Codec
• Streaming Inference
• Factorized Representation
• Low-Latency Speech Processing