Learn2Chat: Rethinking Dyadic Talking Heads via Interaction-Modulated Monologic Priors

2026-07-11

AI Summary

Existing methods for two-person conversational gesture generation struggle to disentangle speech-driven motions from interactive feedback and fail to effectively leverage the speech-to-gesture priors embedded in pretrained single-speaker talking-head models. This work proposes Learn2Chat, a framework that models dyadic behavior as an interaction-modulated adaptation of single-speaker priors through anchor-based motion decomposition and cross-attention-driven prediction of interaction-specific latent variables. The approach explicitly separates speech-driven and interaction-specific components, enabling model-agnostic reuse of pretrained priors and efficient adaptation to dyadic settings. Evaluated on the DualTalk benchmark, Learn2Chat achieves state-of-the-art performance, significantly outperforming existing methods in both quantitative metrics and subjective evaluations, while seamlessly integrating diverse single-speaker motion generation backbones.

Abstract

Dyadic conversational motion generation is essential for realistic interactive digital humans. Existing approaches typically model conversational behaviors within unified dyadic generators. However, such holistic formulations tend to couple self-speech-driven motion with partner-responsive social feedback, leaving the interaction-specific component implicit and underutilizing the speech-motion correspondence already learned by pretrained monologic motion models. We propose Learn2Chat, a unified framework that models dyadic motion as interaction modulation over pretrained monologic motion priors. This design separates intrinsic speech-driven motion from social interaction effects and enables more structured interaction modeling. Specifically, we introduce a Monologic-Anchored Motion Factorization scheme that leverages the semantic motion manifold learned from monologic data to disentangle audio-driven motion dynamics from interaction-induced modulation, yielding clean interaction representations from dyadic sequences. On top of this representation space, a Cross-Attentive Interaction Latent Prediction module maps paired speech signals to interaction latents through cross-branch attention and interaction alignment. During inference, the predicted interaction latents modulate canonical monologic motion to generate coherent and synchronized dyadic behaviors in a data-efficient manner. Extensive experiments on the DualTalk benchmark demonstrate that Learn2Chat achieves state-of-the-art performance across both quantitative metrics and perceptual evaluations. Moreover, the framework is model-agnostic and seamlessly integrates with diverse pretrained monologic motion backbones, highlighting the effectiveness of prior reuse and interaction adaptation for scalable conversational motion generation. More visual results are available on the project page.

Problem

• dyadic motion generation
• monologic priors
• interaction modulation
• speech-motion correspondence
• conversational behavior

Innovation

• interaction modulation
• monologic motion priors
• motion factorization
• cross-attentive latent prediction
• dyadic talking heads