AU Codes, Language, and Synthesis: Translating Anatomy to Text for Facial Behavior Synthesis

2026-03-19

AI Summary

Existing facial behavior synthesis methods rely on coarse-grained emotion labels or linear combinations of action units (AUs), struggling to model anatomically plausible complex expressions and often yielding unnatural results. This work proposes a text-guided, high-fidelity facial expression generation framework by first translating AUs into natural language descriptions. The key contributions include constructing BP4D-AUText, the first large-scale dataset of paired AUs and textual descriptions; designing a rule-based dynamic AU text processor; and introducing VQ-AUFace, a generative model that integrates facial structural priors with modern text-to-image techniques to enhance anatomical plausibility. Experimental results demonstrate that the proposed approach significantly outperforms existing methods in both quantitative metrics and user studies, particularly excelling at synthesizing realistic, expressive, and perceptually credible facial behaviors when handling conflicting AUs.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

Facial behavior synthesis remains a critical yet underexplored challenge. While text-to-face models have made progress, they often rely on coarse emotion categories, which lack the nuance needed to capture the full spectrum of human nonverbal communication. Action Units (AUs) provide a more precise and anatomically grounded alternative. However, current AU-based approaches typically encode AUs as one-hot vectors, modeling compound expressions as simple linear combinations of individual AUs. This linearity becomes problematic when handling conflicting AUs--defined as those which activate the same facial muscle with opposing actions. Such cases lead to anatomically implausible artifacts and unnatural motion superpositions. To address this, we propose a novel method that represents facial behavior through natural language descriptions of AUs. This approach preserves the expressiveness of the AU framework while enabling explicit modeling of complex and conflicting AUs. It also unlocks the potential of modern text-to-image models for high-fidelity facial synthesis. Supporting this direction, we introduce BP4D-AUText, the first large-scale text-image paired dataset for complex facial behavior. It is synthesized by applying a rule-based Dynamic AU Text Processor to the BP4D and BP4D+ datasets. We further propose VQ-AUFace, a generative model that leverages facial structural priors to synthesize realistic and diverse facial behaviors from text. Extensive quantitative experiments and user studies demonstrate that our approach significantly outperforms existing methods. It excels in generating facial expressions that are anatomically plausible, behaviorally rich, and perceptually convincing, particularly under challenging conditions involving conflicting AUs.

Problem

• Facial Behavior Synthesis
• Action Units
• Conflicting AUs
• Anatomical Plausibility
• Nonverbal Communication

Innovation

• Action Units
• Facial Behavior Synthesis
• Natural Language Representation
• Conflicting AUs
• Text-to-Face Generation