DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection

2026-06-28

AI Summary

This work addresses the significant bit overhead incurred by transmitting side information for retained timesteps in variable frame rate speech coding, which limits performance gains under strict total bitrate constraints. To mitigate this, the authors propose a dynamic token masking mechanism: the encoder selectively retains critical tokens while filling other positions with a learned <MASK> embedding, transmitting only a lightweight binary mask to enable position-aware decoding. Additionally, a Path Length Equalization (PLE) linear-time boundary selection algorithm is introduced to generate uniformly distributed adaptive segments, substantially reducing side information overhead. Under strictly matched total bitrate conditions, the proposed method consistently outperforms fixed frame rate baselines across multiple operating points, achieving higher speech reconstruction quality and intelligibility while maintaining minimal side information cost.

Abstract

Variable frame rate (VFR) coding has recently emerged in neural speech codecs, allocating fewer frames to redundant regions and more frames to rapidly changing speech. VFR must transmit side information about retained time steps, but prior gains are either not rigorously addressed or often minor once these overhead bits are included in total bitrate. We present Dynamic Token Masking (DTM)-Codec, a neural speech codec that demonstrates clear gains over fixed-frame-rate baselines under a strict matched-total-bitrate protocol. DTM keeps selected encoder tokens, fills masked positions with a learned <MASK> embedding, and transmits a binary keep-mask for position-aware decoding. We further introduce Path Length Equalization (PLE), a linear-time boundary selector for VFR coding that yields well-spread adaptive segments with negligible overhead. Across operating points, DTM-Codec broadly improves reconstruction quality and intelligibility over fixed-frame-rate baselines.

Problem

• Variable Frame Rate
• Speech Coding
• Bitrate Overhead
• Neural Codec
• Token Masking

Innovation

• Dynamic Token Masking
• Variable Frame Rate
• Path Length Equalization
• Neural Speech Codec
• Binary Keep-Mask