OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation

2025-01-31

AI Summary

Existing 3D dynamic generation methods rely on single-material assumptions, limiting their ability to faithfully model diverse material behaviors—including elasticity, viscoelasticity, plasticity, and fluid dynamics—as well as their complex intermixtures. This work introduces the first general-purpose physics-aware 3D Gaussian scene modeling framework for dynamic generation: it employs 3D Gaussians as primitive representations, each associated with a continuous, learnable weighting over twelve expert physical submodels, enabling compositional and physically interpretable material representation. We integrate a video diffusion model to guide the learning of material weights and support text-driven, cross-material scene generation. Quantitatively, our method surpasses state-of-the-art approaches by 3–16% in both visual quality and text-alignment metrics. Qualitatively, it successfully synthesizes physically plausible dynamic sequences involving intricate material interactions—e.g., metal-water splashing and rubber-honey deformation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects. A key design of OmniPhysGS is treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that OmniPhysGS achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.

Problem

• 3D animation
• physical properties
• material dynamics

Innovation

• 3D Mathematical Modeling
• Multi-Material Dynamics
• Advanced Physical Simulation