CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditions

2026-02-02

AI Summary

This work addresses the challenge of unsupervised learning of cloth dynamics from multi-view visual observations without prior knowledge of physical parameters. The authors propose Cloth Dynamics Splatting (CloDS), an end-to-end framework that integrates three stages: mesh-based Gaussian splatting for geometry reconstruction, video-geometry alignment, and unsupervised dynamics modeling. A key innovation is the introduction of the Cloth Dynamics Grounding setting along with a dual-position opacity modulation mechanism, which establishes a bidirectional mapping between 2D observations and 3D geometry, effectively mitigating issues caused by large deformations and self-occlusions. Experiments demonstrate that CloDS accurately simulates complex cloth dynamics using only visual inputs—without any physical priors—and exhibits strong generalization capabilities.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Deep learning has demonstrated remarkable capabilities in simulating complex dynamic systems. However, existing methods require known physical properties as supervision or inputs, limiting their applicability under unknown conditions. To explore this challenge, we introduce Cloth Dynamics Grounding (CDG), a novel scenario for unsupervised learning of cloth dynamics from multi-view visual observations. We further propose Cloth Dynamics Splatting (CloDS), an unsupervised dynamic learning framework designed for CDG. CloDS adopts a three-stage pipeline that first performs video-to-geometry grounding and then trains a dynamics model on the grounded meshes. To cope with large non-linear deformations and severe self-occlusions during grounding, we introduce a dual-position opacity modulation that supports bidirectional mapping between 2D observations and 3D geometry via mesh-based Gaussian splatting in video-to-geometry grounding stage. It jointly considers the absolute and relative position of Gaussian components. Comprehensive experimental evaluations demonstrate that CloDS effectively learns cloth dynamics from visual data while maintaining strong generalization capabilities for unseen configurations. Our code is available at https://github.com/whynot-zyl/CloDS. Visualization results are available at https://github.com/whynot-zyl/CloDS_video}.%\footnote{As in this example.

Problem

• cloth dynamics
• unsupervised learning
• visual-only observation
• unknown conditions
• multi-view

Innovation

• unsupervised learning
• cloth dynamics
• Gaussian splatting
• multi-view vision
• 3D reconstruction