Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation

2026-06-29

AI Summary

This work addresses the challenge of achieving flexibility, natural motion, and real-time performance in conversational talking-face generation involving multi-turn, multi-participant interactions. To this end, we propose InterTalk, a motion-driven framework that explicitly models conversational dynamics among multiple speakers. InterTalk introduces decoupled facial motion control—separating lip movements, blinking, and other expressions—and employs an iterative generation strategy enhanced by multi-source motion feedback and 3D data augmentation. For the first time, our approach enables unified, real-time (30 FPS) synthesis of highly natural talking faces with arbitrary numbers of participants across multiple dialogue turns. Extensive experiments on a newly collected large-scale multi-speaker conversational dataset demonstrate that InterTalk significantly outperforms existing methods in terms of interaction realism, generation efficiency, and overall flexibility.

Abstract

Conversational talking face generation has recently attracted increasing attention, aiming to synthesize interactive talking videos where characters speak, listen, and respond dynamically to each other. This task presents three core challenges: 1) Flexibility: enabling multi-round dialogues with an arbitrary number of participants; 2) Naturalness: maintaining coherent motion and appropriate non-verbal feedback throughout the interaction; and 3) Efficiency: achieving real-time generation and low computation overhead for long-term continuous online conversation. Despite recent advances, existing methods still fall short in balancing all three requirements. To bridge this gap, we introduce InterTalk, a novel and efficient framework designed for highly interactive conversational talking face generation. Built upon a motion-based architecture, InterTalk supports real-time conversation synthesis. Our method achieves strong flexibility by explicitly modeling multi-round conversational dynamics among each participant, eliminating constraints on their numbers. To enhance interactivity, we incorporate motion feedback from multiple participants and introduce an iterative generation strategy for more natural behaviors. Besides, we disentangle motion into several facial components, enabling targeted refinements for natural response such as precise lip sync and realistic eye blinking. Finally, we construct a new multi-person conversational dataset and enrich it with 3D face-based data augmentation. Extensive experiments demonstrate that InterTalk achieves superior interaction quality while maintaining real-time performance at 30 FPS.

Problem

• Conversational Talking Face Generation
• Flexibility
• Naturalness
• Efficiency
• Interactive Video Synthesis

Innovation

• Conversational Talking Face
• Real-time Generation
• Multi-person Interaction
• Motion Disentanglement
• Iterative Generation