EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

2026-07-02

AI Summary

Existing text-to-3D face generation methods struggle to achieve fine-grained expression control due to the joint modeling of identity, expression, and texture. This work addresses this limitation by decoupling the task into a semantic regression problem within the 3D Morphable Model (3DMM) parameter space. We introduce Txt2Emote, the first fine-grained text–3D expression paired dataset, and propose EmoteGPT, a novel framework that leverages a multimodal large language model augmented with a dedicated <Expr> token to enable precise mapping from text to disentangled 3DMM expression parameters. Experiments demonstrate that our approach significantly outperforms existing methods in both emotion recognition accuracy and perceptual expressiveness, enabling high-fidelity and stylized 3D avatar generation as well as 3D-consistent 2D facial expression synthesis.

Abstract

Precise control of 3D facial expressions from text is crucial for virtual avatars, animation, and human-computer interaction, yet existing text-to-3D methods jointly generate identity, expression, and texture, making fine-grained expression control difficult. We instead formulate text-driven expression synthesis as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). This setting, however, requires paired data linking detailed language to precise expression parameters, which are missing from existing resources. To fill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptions detailing facial features and implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model (MLLM) with a dedicated <Expr> token to semantically ground expression representations, which are then decoded into 3DMM parameters. We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods on emotion recognition metrics and in perceived expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.

Problem

• text-to-3D
• facial expression
• 3D Morphable Model
• expression control
• natural language

Innovation

• text-to-3D expression
• 3D Morphable Model
• disentangled representation
• multimodal LLM
• facial animation