M-PhyGs: Multi-Material Object Dynamics from Video

2025-12-18

AI Summary

This work addresses the challenging problem of estimating physical parameters for real-world multi-material, geometrically complex objects—exemplified by flowers—beyond conventional assumptions of single-material composition, rigidity, and pre-defined dynamical models. We propose the first end-to-end video-driven framework that jointly performs multi-material segmentation and continuous-medium mechanical parameter estimation (i.e., Young’s modulus and Poisson’s ratio) using a multi-material Gaussian radiance field. Our method integrates differentiable physics-based simulation, spatiotemporal consistency constraints, cascaded 3D/2D geometry-appearance losses, and temporal mini-batch optimization. To support systematic evaluation, we introduce Phlowers—the first benchmark dataset for flower–environment interaction. On Phlowers, our approach reduces average error in both multi-material segmentation and physical parameter estimation by 37% over baseline methods, significantly enhancing generalization under dynamic interactions.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Knowledge of the physical material properties governing the dynamics of a real-world object becomes necessary to accurately anticipate its response to unseen interactions. Existing methods for estimating such physical material parameters from visual data assume homogeneous single-material objects, pre-learned dynamics, or simplistic topologies. Real-world objects, however, are often complex in material composition and geometry lying outside the realm of these assumptions. In this paper, we particularly focus on flowers as a representative common object. We introduce Multi-material Physical Gaussians (M-PhyGs) to estimate the material composition and parameters of such multi-material complex natural objects from video. From a short video captured in a natural setting, M-PhyGs jointly segments the object into similar materials and recovers their continuum mechanical parameters while accounting for gravity. M-PhyGs achieves this efficiently with newly introduced cascaded 3D and 2D losses, and by leveraging temporal mini-batching. We introduce a dataset, Phlowers, of people interacting with flowers as a novel platform to evaluate the accuracy of this challenging task of multi-material physical parameter estimation. Experimental results on Phlowers dataset demonstrate the accuracy and effectiveness of M-PhyGs and its components.

Problem

• Estimates material composition and mechanical parameters from video
• Segments objects into similar materials and recovers continuum parameters
• Focuses on multi-material complex natural objects like flowers

Innovation

• Multi-material segmentation from video
• Recovering continuum mechanical parameters
• Cascaded 3D and 2D losses with temporal mini-batching