MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild

2026-06-15

AI Summary

Existing multi-party turn-taking models rely on complex sensor setups, hindering their deployment in real-world human–robot interaction scenarios. This work proposes MuVAP, a causal multimodal framework that operates solely on single-channel audio and single-view video. MuVAP associates speech activity with speakers through facial trajectories and introduces a role-relative projection mechanism to uniformly model any N-person conversation as a fixed state between the current and next speaker. To support this research, we construct the first unedited, single-view audiovisual corpus of multi-party conversations, comprising 31 hours of natural dialogue. Experiments demonstrate that MuVAP significantly outperforms strong baselines on both turn-hold/turn-switch classification and next-speaker prediction tasks in two- and three-party conversations.

Abstract

Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.

Problem

• multiparty turn-taking
• voice activity projection
• human-robot interaction
• audiovisual dataset
• speaker prediction

Innovation

• Multimodal Turn-taking
• Voice Activity Projection
• Role-Relative Projection
• Monaural Audio
• Unedited Conversation Dataset