FRESA:Feedforward Reconstruction of Personalized Skinned Avatars from Few Images

2025-03-24

AI Summary

This work addresses the challenge of real-time, animatable 3D personalized human avatar reconstruction from only a few monocular images—overcoming key limitations of existing methods, including reliance on per-sample optimization and poor generalization across diverse body shapes, poses, and clothing. We propose an end-to-end feedforward network that jointly infers personalized geometry, skinning weights, and pose-dependent deformations—a first in monocular avatar reconstruction. To resolve geometric-skinning ambiguity, we introduce a novel 3D normalization procedure; further, pixel-aligned initialization and multi-frame feature aggregation enhance detail fidelity and identity consistency. Trained on a large-scale clothed human dataset, our method surpasses state-of-the-art approaches in geometric accuracy, animation naturalness, and visual realism. It supports input from smartphone-captured images, achieves millisecond-level inference latency, and enables zero-shot generalization—facilitating practical deployment.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Learning with Manifolds

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We present a novel method for reconstructing personalized 3D human avatars with realistic animation from only a few images. Due to the large variations in body shapes, poses, and cloth types, existing methods mostly require hours of per-subject optimization during inference, which limits their practical applications. In contrast, we learn a universal prior from over a thousand clothed humans to achieve instant feedforward generation and zero-shot generalization. Specifically, instead of rigging the avatar with shared skinning weights, we jointly infer personalized avatar shape, skinning weights, and pose-dependent deformations, which effectively improves overall geometric fidelity and reduces deformation artifacts. Moreover, to normalize pose variations and resolve coupled ambiguity between canonical shapes and skinning weights, we design a 3D canonicalization process to produce pixel-aligned initial conditions, which helps to reconstruct fine-grained geometric details. We then propose a multi-frame feature aggregation to robustly reduce artifacts introduced in canonicalization and fuse a plausible avatar preserving person-specific identities. Finally, we train the model in an end-to-end framework on a large-scale capture dataset, which contains diverse human subjects paired with high-quality 3D scans. Extensive experiments show that our method generates more authentic reconstruction and animation than state-of-the-arts, and can be directly generalized to inputs from casually taken phone photos. Project page and code is available at https://github.com/rongakowang/FRESA.

Problem

• Reconstruct 3D avatars from few images with realistic animation
• Overcome per-subject optimization limits in existing methods
• Improve geometric fidelity and reduce deformation artifacts

Innovation

• Universal prior learning for instant avatar generation
• Personalized shape, skinning weights, and deformations inference
• 3D canonicalization for fine-grained detail reconstruction