Conversational Human Audio-visual Talking Dialogue Generation

2026-07-02

AI Summary

This work addresses the high cost, time consumption, and ethical challenges associated with collecting large-scale audiovisual dyadic interaction data. To overcome these limitations, the authors propose CHAT, a framework that enables end-to-end generation of diverse, identity-rich two-person conversational clips from a single text prompt, with synchronized and mutually responsive speech and facial dynamics. CHAT unifies a language model with a talking-face generation model and introduces an interactive audiovisual behavior optimization module, achieving, for the first time, direct high-quality synthesis of dyadic interactive audiovisual conversations from text. The resulting CHAT-AVD-50k synthetic dataset significantly enhances the performance of downstream models PerFRDiff and ReactDiff on the REACT 2024 benchmark, outperforming existing methods in both objective metrics and subjective evaluations.

Abstract

Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consuming, expensive, and ethically sensitive. To address this, we propose CHAT, a new dyadic interactive audio-visual dialogue generation (DIADG) framework that generates diverse, paired, and mutually responsive speech-face dialogue clips from a single textual prompt. CHAT unifies large language models and talking face models with interactive audio and facial behaviour refinement modules, enabling the generation of aligned dyadic dialogue clips with diverse contents and facial identities. Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations. Moreover, our synthesised CHAT-AVD-50k dataset serves as effective pre-training data for downstream interactive head generation, consistently improving PerFRDiff and ReactDiff on REACT 2024. CHAT offers a scalable alternative to the costly and ethically sensitive collection of real dyadic interaction data.

Problem

• dyadic interactive audio-visual dialogue
• data collection
• ethical sensitivity
• costly annotation
• virtual agents

Innovation

• dyadic dialogue generation
• talking face synthesis
• audio-visual alignment
• interactive behavior refinement
• synthetic dataset