MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations

2026-06-26

AI Summary

Existing methods for two-person conversational facial animation struggle to simultaneously achieve deep semantic understanding and fine-grained dynamic control, often resulting in animations that lack emotional coherence and naturalness. This work proposes MindFlow, a dual-pathway generative framework inspired by the ventral and dorsal streams in neuroscience, which decouples generation into semantic reasoning and motion control. The ventral pathway introduces a novel Chunk-State modeling approach to capture fine-grained emotional state evolution within utterances, while the dorsal pathway employs a conditional autoregressive flow-matching network with a selective acoustic injection mechanism to produce high-fidelity, noise-robust animations and effectively manage speaker–listener role transitions. Experiments demonstrate that MindFlow significantly outperforms state-of-the-art methods in both semantic appropriateness and motion naturalness, generating more realistic, emotionally coherent, and dynamically precise two-person conversational facial animations.

Abstract

Generating lifelike facial animation for dyadic conversations requires reconciling high-level cognitive intent with precise low-level motor reflexes, yet existing methods fall short in the semantic understanding of dialogue context and in precise dynamic control. In this paper, we propose MindFlow, a dual-pathway generative framework inspired by the Ventral-Dorsal pathway model in neuroscience, which decouples generation into two collaborative streams, thereby harmonizing deep semantic reasoning with fine-grained control. In the Ventral module, we transform the conventional Sentence-Action approach into a novel Chunk-State approach that models raw acoustic streams as a context-aware, evolving emotional state chain, capturing subtle paralinguistic nuances and mid-utterance emotional shifts missed by sentence-level modeling. The Dorsal module features a conditional autoregressive flow matching network for high-fidelity facial motion, driven by high-frequency acoustic cues and modulated by emotion states, plus a Selective Acoustic Injector for adaptive audio gating to ensure robustness in talking-and-listening dynamics without interference. Extensive experiments demonstrate that MindFlow achieves superior semantic appropriateness and motion naturalness compared to state-of-the-art baselines.

Problem

• facial animation generation
• dyadic conversations
• cognitive semantics
• acoustic dynamics
• semantic understanding

Innovation

• dual-pathway generative framework
• Chunk-State modeling
• conditional autoregressive flow matching
• Selective Acoustic Injector
• emotion-aware facial animation