When2Speak: A Dataset for Temporal Participation and Turn-Taking in Multi-Party Conversations for Large Language Models

2026-05-06

AI Summary

This study addresses the challenge of large language models frequently interrupting interlocutors in multi-turn, multi-party dialogues due to inaccurate turn-taking decisions, which undermines conversational coherence. The work introduces speaking timing as a trainable dimension of dialog intelligence and presents the When2Speak dataset. It proposes a four-stage synthetic data generation pipeline that integrates real-world scenarios with structured augmentation. Through supervised fine-tuning (SFT) and reinforcement learning with asymmetric rewards, the approach substantially improves model performance: SFT yields an average 60% increase in Macro F1 score (up to 120% in some cases), while reinforcement learning reduces the missed-interruption rate from 0.50 to 0.186–0.218 and achieves a recall of 0.78–0.81.

Abstract

Large Language Models (LLMs) excel at generating contextually appropriate responses but remain poorly calibrated for multi-party conversations, where deciding when to speak is as critical as what to say. In such settings, naively responding at every turn leads to excessive interruptions and degraded conversational coherence. We introduce When2Speak, a grounded synthetic dataset and four-stage generation pipeline for learning intervention timing in group interactions. The dataset comprises over 215,000 examples derived from 16,000 conversations involving 2-6 speakers, spanning diverse conversational styles, tones, and participant dynamics, and explicitly modeling SPEAK vs. SILENT decisions at each turn. Our pipeline combines real-world grounding, structured augmentation, controlled transcript synthesis, and fine-tuning-ready supervision, and is fully open-sourced to support reproducibility and adaptation to domain-specific conversational norms. Across multiple model families, supervised fine-tuning (SFT) on When2Speak significantly outperforms zero-shot baselines (e.g., the average Macro F1 increase across 4B+ parameter models was 60%, with the largest increase being 120%). However, SFT-trained models remain systematically over-conservative, missing nearly half of warranted interventions as seen through the Missed Intervention Rate (MIR), which was on average 0.50 and is noticed even at larger model sizes. To address this limitation, we apply reinforcement learning with asymmetric reward shaping, which reduces MIR to 0.186-0.218 and increases recall from 0.479 to 0.78-0.81. Our findings establish that temporal participation is a distinct and trainable dimension of conversational intelligence, and that grounded synthetic data provides an effective and scalable pathway for enabling LLMs to participate more naturally and appropriately in multi-party interactions.

Problem

• turn-taking
• multi-party conversations
• temporal participation
• intervention timing
• conversational intelligence

Innovation

• turn-taking
• temporal participation
• synthetic dataset
• reinforcement learning
• multi-party conversation