HumanMaterial: Human Material Estimation from a Single Image via Progressive Training

2025-07-24

AI Summary

This paper addresses the ill-posed problem of full-body material estimation from a single human image. We propose HumanMaterial: first, we introduce OpenHumanBRDF—a high-fidelity, open-source dataset—marking the first explicit modeling of displacement maps and subsurface scattering (SSS) in human inverse rendering; second, we design a progressive multi-stage training framework integrated with controllable physically based rendering (PBR) loss (CPR), jointly optimizing six physically grounded material maps—normal, albedo, roughness, specular, displacement, and SSS. To mitigate multi-task imbalance and underfitting, we incorporate physics-driven prior prediction and fine-tuning. Our method achieves significant improvements over state-of-the-art approaches on both OpenHumanBRDF and real-world images, particularly enhancing geometric fidelity and optical realism in skin regions, enabling high-quality, photorealistic relighting under arbitrary illumination.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Full-body Human inverse rendering based on physically-based rendering aims to acquire high-quality materials, which helps achieve photo-realistic rendering under arbitrary illuminations. This task requires estimating multiple material maps and usually relies on the constraint of rendering result. The absence of constraints on the material maps makes inverse rendering an ill-posed task. Previous works alleviated this problem by building material dataset for training, but their simplified material data and rendering equation lead to rendering results with limited realism, especially that of skin. To further alleviate this problem, we construct a higher-quality dataset (OpenHumanBRDF) based on scanned real data and statistical material data. In addition to the normal, diffuse albedo, roughness, specular albedo, we produce displacement and subsurface scattering to enhance the realism of rendering results, especially for the skin. With the increase in prediction tasks for more materials, using an end-to-end model as in the previous work struggles to balance the importance among various material maps, and leads to model underfitting. Therefore, we design a model (HumanMaterial) with progressive training strategy to make full use of the supervision information of the material maps and improve the performance of material estimation. HumanMaterial first obtain the initial material results via three prior models, and then refine the results by a finetuning model. Prior models estimate different material maps, and each map has different significance for rendering results. Thus, we design a Controlled PBR Rendering (CPR) loss, which enhances the importance of the materials to be optimized during the training of prior models. Extensive experiments on OpenHumanBRDF dataset and real data demonstrate that our method achieves state-of-the-art performance.

Problem

• Estimating high-quality human materials from single images
• Overcoming ill-posed inverse rendering with realistic material constraints
• Balancing multi-material prediction via progressive training strategy

Innovation

• Constructed high-quality dataset OpenHumanBRDF for realistic rendering
• Designed HumanMaterial model with progressive training strategy
• Introduced Controlled PBR Rendering loss for material optimization