PGC: Physics-Based Gaussian Cloth from a Single Pose

2025-03-26

AI Summary

Existing methods struggle to simultaneously achieve high-fidelity garment reconstruction and strong generalization across novel poses and body shapes, while heavily relying on multi-frame input sequences. This paper introduces the first method capable of reconstructing high-fidelity, physically simulatable cloth models from a single static multi-view image capture. Our core innovation is a hybrid mesh-embedded 3D Gaussian representation—combining geometric and material differentiability with high-frequency detail modeling. We integrate differentiable mesh rendering, 3D Gaussian splatting, physics-based material optimization, and neural reflectance modeling. The approach enables high-quality single-frame reconstruction, cross-pose physical simulation, real-time surface rendering, and controllable relighting. By drastically reducing data acquisition and computational overhead, our method overcomes the fundamental limitation of dynamic generalization from static inputs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

We introduce a novel approach to reconstruct simulation-ready garments with intricate appearance. Despite recent advancements, existing methods often struggle to balance the need for accurate garment reconstruction with the ability to generalize to new poses and body shapes or require large amounts of data to achieve this. In contrast, our method only requires a multi-view capture of a single static frame. We represent garments as hybrid mesh-embedded 3D Gaussian splats, where the Gaussians capture near-field shading and high-frequency details, while the mesh encodes far-field albedo and optimized reflectance parameters. We achieve novel pose generalization by exploiting the mesh from our hybrid approach, enabling physics-based simulation and surface rendering techniques, while also capturing fine details with Gaussians that accurately reconstruct garment details. Our optimized garments can be used for simulating garments on novel poses, and garment relighting. Project page: https://phys-gaussian-cloth.github.io .

Problem

• Reconstruct simulation-ready garments with intricate appearance
• Balance accurate reconstruction with pose and shape generalization
• Require minimal data for garment simulation and relighting

Innovation

• Hybrid mesh-embedded 3D Gaussian splats
• Single static frame multi-view capture
• Physics-based simulation with fine details