Adaptive Turn-Taking for Real-time Multi-Party Voice Agents

2026-06-11

AI Summary

This work addresses the challenge of achieving natural and precise turn-taking in multi-party spoken dialogues, where agents struggle with dynamic turn competition and shifting user expectations. The authors propose ModeratorLM, a role-playing voice agent that introduces explicit role conditioning into multi-speaker turn control for the first time. It integrates a chunked streaming speech large language model with a role- and context-aware chain-of-thought reasoning mechanism. To support training and evaluation, they also construct RolePlayConv, a large-scale synthetic multi-party dialogue dataset. Experimental results demonstrate that ModeratorLM improves turn-taking precision by over 40% and recall by more than 70% on both real-world meeting transcripts and the RolePlayConv dataset, while significantly reducing false interruption rates.

Abstract

Turn-taking in multi-party spoken conversations remains a fundamental challenge for voice-based agents, particularly under dynamic floor competition and varying user expectations. We propose ModeratorLM, a role-playing voice agent that conditions turn-taking behavior on an explicitly assigned role in multi-party settings. The system is built on a speech large language model operating in chunk-wise streaming manner. We further introduce a reasoning-augmented variant that incorporates chain-of-thought reasoning over conversational context and the assigned role. We construct RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles. Experiments on real-world meeting data and RolePlayConv show improved turn-taking precision by over 40% and recall by more than 70%, while substantially reducing false-positive interruptions compared to non-role-conditioned baselines.

Problem

• turn-taking
• multi-party conversation
• voice agents
• floor competition
• spoken dialogue

Innovation

• adaptive turn-taking
• role-conditioned voice agent
• speech large language model
• chain-of-thought reasoning
• multi-party conversation