Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactions

2025-11-29

AI Summary

Monocular 3D semantic reconstruction of dynamic multi-person–multi-object interactions in real-world scenes faces challenges including diverse motion patterns, severe occlusions, and temporal inconsistency; existing Gaussian Splatting (GS) methods struggle to preserve structural coherence. This paper proposes a hybrid reconstruction framework integrating generative modeling with GS: it introduces a semantic-aware rigid-body + Linear Blend Skinning (LBS) deformation model, coupled with asset-driven fine-grained mesh mapping, enabling joint geometric and semantic optimization; furthermore, 3D generative priors guide GS optimization to ensure temporally consistent, high-fidelity surface reconstruction under occlusion. Evaluated on the HOI-M3 dataset, our method significantly outperforms state-of-the-art approaches, achieving high-quality, multi-view consistent, and temporally continuous 3D reconstructions of dynamic scenes.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Real-world human-built environments are highly dynamic, involving multiple humans and their complex interactions with surrounding objects. While 3D geometry modeling of such scenes is crucial for applications like AR/VR, gaming, and embodied AI, it remains underexplored due to challenges like diverse motion patterns and frequent occlusions. Beyond novel view rendering, 3D Gaussian Splatting (GS) has demonstrated remarkable progress in producing detailed, high-quality surface geometry with fast optimization of the underlying structure. However, very few GS-based methods address multihuman, multiobject scenarios, primarily due to the above-mentioned inherent challenges. In a monocular setup, these challenges are further amplified, as maintaining structural consistency under severe occlusion becomes difficult when the scene is optimized solely based on GS-based rendering loss. To tackle the challenges of such a multihuman, multiobject dynamic scene, we propose a hybrid approach that effectively combines the advantages of 1) 3D generative models for generating high-fidelity meshes of the scene elements, 2) Semantic-aware deformation, ie rigid transformation of the rigid objects and LBS-based deformation of the humans, and mapping of the deformed high-fidelity meshes in the dynamic scene, and 3) GS-based optimization of the individual elements for further refining their alignments in the scene. Such a hybrid approach helps maintain the object structures even under severe occlusion and can produce multiview and temporally consistent geometry. We choose HOI-M3 for evaluation, as, to the best of our knowledge, this is the only dataset featuring multihuman, multiobject interactions in a dynamic scene. Our method outperforms the state-of-the-art method in producing better surface reconstruction of such scenes.

Problem

• Reconstructs dynamic scenes with multiple human-object interactions.
• Addresses occlusion and motion challenges in monocular 3D modeling.
• Combines generative models and Gaussian Splatting for consistent geometry.

Innovation

• Hybrid approach combining 3D generative models and Gaussian Splatting
• Semantic-aware deformation for rigid objects and human models
• Maintains structural consistency under occlusion via mesh mapping