Next-Turn: Duration-Aware Streaming Endpoint Detection via Time-to-Next-Speech-Onset Prediction

2026-06-16

AI Summary

In streaming speech systems, speaker hesitations and disfluencies pose significant challenges for semantic endpoint detection due to ambiguous supervision and stringent latency constraints. This work proposes using the time until the start of the next speech segment as a self-supervised training target, leveraging raw audio timestamps to generate precise supervisory signals without requiring additional annotations, thereby enabling duration-aware modeling of pauses. The approach introduces continuous duration prediction into endpoint detection for the first time and jointly trains it with conventional binary classification. Experimental results demonstrate that, under a 320-millisecond latency budget, the proposed method achieves an absolute improvement of 25.9% in endpoint detection accuracy over the strongest baseline, with performance gains consistently increasing as pause duration grows.

Abstract

Endpoint detection (EPD) is essential for natural turn-taking in streaming speech systems. However, reliably determining the endpoint of an utterance is challenging because speakers often pause mid-utterance due to hesitations and disfluencies. Semantic EPD has emerged as a promising direction to address this issue but is hindered by ambiguous supervision and strict streaming constraints. We propose Next-Turn that uses the time-to-next-speech-onset as the training objective, where targets are derived directly from speech timestamps and require no additional annotation. Experiments show that the proposed method outperforms conventional acoustic and recent semantic EPD baselines, achieving a 25.9% absolute improvement in endpoint accuracy within 320 ms over the strongest baseline. In addition, joint training with the duration-aware objective complements standard binary EPD, with gains that increase monotonically with increasing pauses.

Problem

• endpoint detection
• streaming speech
• turn-taking
• speech disfluency
• utterance boundary

Innovation

• endpoint detection
• time-to-next-speech-onset
• streaming speech
• turn-taking
• duration-aware