Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection

2026-09-13

AI Summary

本文研究视觉特征如何改善基于音频的轮流发言预测，通过扩展VAP模型并融合面部动作、视线方向等视觉信息，结果表明这些信息能显著提高性能。

Abstract

Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face conversations. The visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU). For incorporating the visual features, we explore concatenation, cross-attention fusion, delta features, and trainable gating mechanisms. Results show that visual information improves performance over the audio-only baseline, with FAU being significantly more informative than other feature groups. Body and gaze features nevertheless contribute complementary information, as the model combining all features performs best. Furthermore, results indicate that performance on specific tasks varies depending on whether training and test data come from improvised (acted) or naturalistic (non-acted) conversations.

Problem

• turn-taking
• multimodal cues
• face-to-face conversation
• visual features
• voice activity projection

Innovation

• multimodal turn-taking
• visual features
• Voice Activity Projection (VAP)
• facial action units (FAU)
• cross-attention fusion