CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-Consistency from a Single Image

2025-03-19

AI Summary

Single-image reconstruction of clothed humans under heavy occlusion suffers from multi-view inconsistency and geometric fragmentation, while existing approaches heavily rely on hard-to-obtain ground-truth geometric priors (e.g., SMPL). This paper proposes the first 3D-supervision-free and geometry-annotation-free two-stage framework. In Stage I, a controllable pose-guided multi-view diffusion model synthesizes robust, occlusion-aware novel views. In Stage II, conditioned on these synthesized views, the framework predicts cross-view-aligned 3D Gaussian representations and introduces an unsupervised cross-view alignment loss to jointly optimize occlusion robustness and geometric consistency. Evaluated under severe occlusion, our method significantly improves novel-view synthesis quality (+3 dB PSNR) and 3D reconstruction accuracy while preserving structural integrity—overcoming long-standing reconstruction bottlenecks in unconstrained, heavily occluded real-world scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Reconstructing clothed humans from a single image is a fundamental task in computer vision with wide-ranging applications. Although existing monocular clothed human reconstruction solutions have shown promising results, they often rely on the assumption that the human subject is in an occlusion-free environment. Thus, when encountering in-the-wild occluded images, these algorithms produce multiview inconsistent and fragmented reconstructions. Additionally, most algorithms for monocular 3D human reconstruction leverage geometric priors such as SMPL annotations for training and inference, which are extremely challenging to acquire in real-world applications. To address these limitations, we propose CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-ConsistEncy from a Single Image, a novel pipeline designed to reconstruct occlusion-resilient 3D humans with multiview consistency from a single occluded image, without requiring either ground-truth geometric prior annotations or 3D supervision. Specifically, CHROME leverages a multiview diffusion model to first synthesize occlusion-free human images from the occluded input, compatible with off-the-shelf pose control to explicitly enforce cross-view consistency during synthesis. A 3D reconstruction model is then trained to predict a set of 3D Gaussians conditioned on both the occluded input and synthesized views, aligning cross-view details to produce a cohesive and accurate 3D representation. CHROME achieves significant improvements in terms of both novel view synthesis (upto 3 db PSNR) and geometric reconstruction under challenging conditions.

Problem

• Reconstructing clothed humans from single occluded images.
• Achieving multiview consistency without geometric priors.
• Improving 3D human reconstruction under challenging conditions.

Innovation

• Multiview diffusion model synthesizes occlusion-free images.
• 3D Gaussians predict accurate 3D human reconstructions.
• No ground-truth geometric prior annotations required.