StreamAlign: Streaming Text-Aligned Speech Tokenization

2026-09-09

AI Summary

为解决实时语音文本联合建模中的词汇不匹配和延迟问题，StreamAlign通过结合字符级对齐与词级ASR指导实现了流式文本对齐语音分词。

Abstract

Text-aligned speech tokenization methods have emerged to better align speech tokens with LLM token spaces, enabling more effective utilization of pretrained LLMs. However, they rely on offline automatic speech recognition (ASR), leading to two key limitations: (i) the need for complete utterances before tokenization, precluding real-time streaming, and (ii) vocabulary mismatch between ASR and LLMs, which reduces acoustic granularity from the subword to the word level. We introduce StreamAlign, a text-aligned speech tokenization framework that enables streaming tokenization for real-time speech-text joint modeling. StreamAlign performs online speech-text alignment by combining character-level RNN-Transducer alignment with word-level ASR guidance, mitigating ASR-LLM vocabulary mismatch while preserving recognition accuracy. A proactive word boundary classifier anticipates word completion at chunk boundaries, reducing tokenization latency from 560 ms to 270 ms. On LibriSpeech, StreamAlign achieves the lowest WER and highest UTMOS among evaluated tokenizers. Furthermore, StreamAlign-SLM, a spoken language model trained on StreamAlign units, outperforms other end-to-end spoken language models in speech continuation while achieving the strongest overall consistency on SALMon and spoken StoryCloze.

Problem

• text-aligned speech tokenization
• offline automatic speech recognition
• vocabulary mismatch

Innovation

• Streaming Tokenization
• RNN-Transducer
• Word Boundary Classifier
• Real-time Speech-Text Joint Modeling
• Vocabulary Mismatch Mitigation