Fast When, Careful Who: Dual-Process Multiparty Turn-Taking with Diffusion Augmentation

2026-06-15

AI Summary

This work addresses the challenge of accurately modeling turn-taking in multi-party conversations, where overlapping speech and rapid speaker switches complicate dialogue understanding. The authors propose a purely audio-based, two-stage pipeline: a fast trigger first identifies candidate turn boundaries, followed by a lightweight verifier that determines whether the current speaker retains the floor or yields it, while also predicting the next speaker. A key innovation lies in decoupling turn-boundary detection from turn-holding decisions and introducing a label-preserving diffusion model to synthesize background-augmented training data. Evaluated on the VoxConverse dataset, the method significantly improves turn-yield detection performance, with the diffusion-based data augmentation strategy providing further measurable gains.

Abstract

Reliable turn-taking is essential for spoken dialogue systems. However, most existing methods are designed for two-speaker interaction and struggle with realistic multiparty audio containing overlap and rapid speaker changes. We study multiparty turn-taking on the VoxConverse dataset and propose an audio-only two-stage pipeline that separates when to trigger a turn boundary from whether the floor is actually transferring. A fast trigger scans the audio and proposes candidate end-of-turn times, while a lightweight verifier runs only at those times to decide \textsc{Hold} or \textsc{Shift} and support next-speaker prediction. We report results in the full multiparty setting and a controlled dyadic top-2 projection for comparability. We also investigate diffusion-based, label-preserving background-audio mixing as a data augmentation strategy. Results show improved shift detection over a baseline, with further improvements from diffusion augmentation.

Problem

• multiparty turn-taking
• speaker overlap
• rapid speaker changes
• spoken dialogue systems
• turn boundary detection

Innovation

• dual-process turn-taking
• multiparty dialogue
• diffusion augmentation
• audio-only pipeline
• floor transfer detection