Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

2026-07-08

AI Summary

This study addresses the challenge of accurately predicting turn-taking timing in mediated human–robot interaction. To this end, the authors propose the Multimodal Voice Activity Projection (MM-VAP) framework, which extends self-supervised voice activity projection to audiovisual modalities for the first time. The approach leverages pretrained encoders enhanced with low-rank adaptation (LoRA) for efficient turn-taking prediction, incorporating a cross-speaker attention mechanism and a semantic consistency loss to model high-order conversational dynamics within a 256-dimensional output space. Evaluated on the NoXi, NoXi+J, and Haru EDR datasets, MM-VAP significantly outperforms existing baselines, demonstrating particularly strong performance in predicting critical turn-taking events and validating its effectiveness in mediated human–robot dialogue scenarios.

Abstract

Turn-taking prediction is a key requirement for social robots involved in human-human interaction, particularly in mediator settings, where the robot must anticipate conversational dynamics rather than merely react to pauses. This work presents a Multimodal Voice Activity Projection (MM-VAP) framework that extends the original audio-only VAP formulation to synchronized audio-visual inputs while preserving its self-supervised future-projection objective. The proposed approach builds on pretrained audio-visual backbones originally optimized for speech-related tasks and adapts them through Low-Rank Adaptation to the multimodal turn-taking problem. After independent speaker encoding, an inter-speaker attention stage models the relational dynamics required to project future voice activity. In addition, a semantic consistency loss is introduced to regularize the 256-state output space according to higher-level dialogue activity patterns. Experiments on NoXi and NoXi+J showed improvements over the current baselines, particularly for some turn-taking events. Additional evaluation on the Haru EDR corpus further supported the suitability of this direction for mediation-oriented human-robot interaction.

Problem

• turn-taking prediction
• social robots
• voice activity projection
• human-robot interaction
• multimodal

Innovation

• Multimodal Voice Activity Projection
• Low-Rank Adaptation
• Turn-Taking Prediction
• Inter-Speaker Attention
• Semantic Consistency Loss