MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning

2025-02-04

AI Summary

NeRF-based digital humans suffer from catastrophic forgetting, color leakage, and pose distortion when continuously learning new appearances and poses. To address these issues, this paper proposes an online-updatable, forgetting-resistant neural radiance field (NeRF) modeling method. Our approach comprises: (1) a global-local joint feature memory module that decouples and preserves appearance representations across tasks; and (2) a pose distillation module that enforces consistency between old and new pose distributions to mitigate pose drift. By integrating continual learning, feature disentanglement, and knowledge distillation, our method achieves high-fidelity rendering of historical appearances with only minimal fine-tuning on new data. It is the first to systematically resolve color leakage and pose distortion in continual learning for NeRF-based digital humans. Experiments demonstrate an average 2.1 dB PSNR improvement on retained scenes, a 37% reduction in color error, and a 29% decrease in pose keypoint error.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The generation of a virtual digital avatar is a crucial research topic in the field of computer vision. Many existing works utilize Neural Radiance Fields (NeRF) to address this issue and have achieved impressive results. However, previous works assume the images of the training person are available and fixed while the appearances and poses of a subject could constantly change and increase in real-world scenarios. How to update the human avatar but also maintain the ability to render the old appearance of the person is a practical challenge. One trivial solution is to combine the existing virtual avatar models based on NeRF with continual learning methods. However, there are some critical issues in this approach: learning new appearances and poses can cause the model to forget past information, which in turn leads to a degradation in the rendering quality of past appearances, especially color bleeding issues, and incorrect human body poses. In this work, we propose a maintainable avatar (MaintaAvatar) based on neural radiance fields by continual learning, which resolves the issues by utilizing a Global-Local Joint Storage Module and a Pose Distillation Module. Overall, our model requires only limited data collection to quickly fine-tune the model while avoiding catastrophic forgetting, thus achieving a maintainable virtual avatar. The experimental results validate the effectiveness of our MaintaAvatar model.

Problem

• Maintain virtual avatar appearance changes
• Avoid forgetting past appearance data
• Update avatar without quality degradation

Innovation

• Neural Radiance Fields
• Continual Learning
• Global-Local Joint Storage