Real-time Generation of Listener Nodding via Prediction of Kinematic Parameters for Avatar Dialogue Systems

2026-07-14

AI Summary

This work proposes a lightweight, real-time model to enhance the human-likeness and interaction fluency of virtual avatars in human-computer dialogue by generating context-aware, natural head-nodding behaviors. The approach employs a dual-channel attention network built upon Voice Activity Projection (VAP) to jointly predict nod timing and kinematic parameters, enabling the first real-time, context-sensitive synthesis of complete nodding gestures—including both onset timing and motion morphology. A novel design initializes the kinematic module with outputs from the timing prediction module, significantly improving gesture naturalness through fine-tuning. Subjective evaluations demonstrate that the proposed method substantially outperforms baseline strategies employing random or fixed nodding motions and has been successfully integrated into a real-world conversational avatar system.

Abstract

In human dialogue, we achieve smooth communication by expressing nonverbal cues such as eye contact, nodding, and facial expressions with precise timing. It is expected for conversational avatars to express these cues appropriately to realize natural and human-like interactions. This study focuses on nodding, which is crucial for demonstrating active listening and encouraging further user utterances. We propose a model that predicts both timing and kinematic parameters representing the motion features of listener nodding in real time. The proposed model consists of a timing prediction module and a kinematic parameter prediction module. Each implements a dyadic attention network over the speaker and listener channels based on the technique of Voice Activity Projection (VAP). Unlike conventional models, this approach enables real-time prediction of kinematic parameters based on the specific context of the dialogue rather than just predicting the timing. Furthermore, we demonstrate the effectiveness of fine-tuning the kinematic parameter prediction module initialized from the trained timing prediction module. The proposed model is lightweight and capable of real-time operation, and it has been integrated into an avatar dialogue system. Subjective evaluation experiments shows that our proposed method significantly outperforms both a baseline with stochastic timing and another with fixed-motion nodding. The code and trained models are available at https://github.com/MaAI-Kyoto/MaAI.

Problem

• listener nodding
• real-time generation
• avatar dialogue systems
• nonverbal cues
• kinematic parameters

Innovation

• real-time nodding generation
• kinematic parameter prediction
• dyadic attention network
• Voice Activity Projection (VAP)
• avatar dialogue system