Deep‐Learning‐Based Facial Retargeting Using Local Patches

2024-10-25

Computer graphics forum (Print)

AI Summary

This work addresses the challenge of semantic distortion that arises when transferring real human facial expressions to stylized 3D characters with significantly different facial proportions and topologies. To mitigate this issue, the authors propose an end-to-end, patch-based facial retargeting method that automatically extracts local expressive regions from source videos and integrates them with the target character’s structural and motion constraints to generate semantically consistent animation parameters. The approach innovatively employs a local patch mechanism, incorporating modules for patch extraction, reenactment, and adaptive weight estimation, thereby effectively preserving the original expression semantics. Experimental results demonstrate that the method produces natural, temporally coherent, and semantically accurate facial animations even on highly exaggerated or non-humanoid 3D characters.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

In the era of digital animation, the quest to produce lifelike facial animations for virtual characters has led to the development of various retargeting methods. While the retargeting facial motion between models of similar shapes has been very successful, challenges arise when the retargeting is performed on stylized or exaggerated 3D characters that deviate significantly from human facial structures. In this scenario, it is important to consider the target character's facial structure and possible range of motion to preserve the semantics assumed by the original facial motions after the retargeting. To achieve this, we propose a local patch‐based retargeting method that transfers facial animations captured in a source performance video to a target stylized 3D character. Our method consists of three modules. The Automatic Patch Extraction Module extracts local patches from the source video frame. These patches are processed through the Reenactment Module to generate correspondingly re‐enacted target local patches. The Weight Estimation Module calculates the animation parameters for the target character at every frame for the creation of a complete facial animation sequence. Extensive experiments demonstrate that our method can successfully transfer the semantic meaning of source facial expressions to stylized characters with considerable variations in facial feature proportion.

Problem

• facial retargeting
• stylized characters
• facial animation
• semantic preservation
• 3D character

Innovation

• facial retargeting
• local patches
• deep learning
• stylized characters
• semantic preservation