AU-Blendshape for Fine-grained Stylized 3D Facial Expression Manipulation

2025-07-16

AI Summary

Addressing the scarcity of high-quality annotated datasets for fine-grained, cross-identity stylized 3D facial expression manipulation, this work introduces AUBlendSet—the first continuous AU-driven 3D facial animation dataset supporting arbitrary identities. It comprises 500 distinct identities, 32 standardized Facial Action Units (AUs), and precise pose annotations. To leverage this resource, we propose AUBlendNet, a novel network based on AU-Blendshape representation that learns identity-invariant AU basis vectors and enables style-adaptive prediction, facilitating parallelized stylized expression generation. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods in three key tasks: stylized facial expression control, speech-driven affective animation, and emotion recognition data augmentation. These results validate substantial advances in generalizability, controllability, and practical applicability—both for the proposed dataset and the model architecture.

Technology Category

• Computer Vision: 3D Computer Vision
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

While 3D facial animation has made impressive progress, challenges still exist in realizing fine-grained stylized 3D facial expression manipulation due to the lack of appropriate datasets. In this paper, we introduce the AUBlendSet, a 3D facial dataset based on AU-Blendshape representation for fine-grained facial expression manipulation across identities. AUBlendSet is a blendshape data collection based on 32 standard facial action units (AUs) across 500 identities, along with an additional set of facial postures annotated with detailed AUs. Based on AUBlendSet, we propose AUBlendNet to learn AU-Blendshape basis vectors for different character styles. AUBlendNet predicts, in parallel, the AU-Blendshape basis vectors of the corresponding style for a given identity mesh, thereby achieving stylized 3D emotional facial manipulation. We comprehensively validate the effectiveness of AUBlendSet and AUBlendNet through tasks such as stylized facial expression manipulation, speech-driven emotional facial animation, and emotion recognition data augmentation. Through a series of qualitative and quantitative experiments, we demonstrate the potential and importance of AUBlendSet and AUBlendNet in 3D facial animation tasks. To the best of our knowledge, AUBlendSet is the first dataset, and AUBlendNet is the first network for continuous 3D facial expression manipulation for any identity through facial AUs. Our source code is available at https://github.com/wslh852/AUBlendNet.git.

Problem

• Lack of datasets for fine-grained 3D facial expression manipulation
• Need for AU-based blendshape representation across diverse identities
• Absence of networks for continuous stylized facial AU manipulation

Innovation

• AUBlendSet dataset for fine-grained expression manipulation
• AUBlendNet learns AU-Blendshape basis vectors
• Parallel prediction for stylized facial manipulation