Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction

2025-11-18

AI Summary

This work addresses the problem of reconstructing dynamic hand–object interactions without object priors. To tackle severe occlusions, ambiguous boundaries, and non-rigid deformations, we propose a novel method that jointly models geometry and appearance via an interaction-aware 4D Gaussian lattice representation. Our approach introduces a dynamic deformation field grounded in a piecewise-linear assumption and integrates explicit regularization constraints with a progressive optimization strategy. Crucially, the framework operates without predefined object models or pose priors, enabling high-fidelity reconstruction of tightly contacting hand–object regions. Evaluated on standard benchmarks, our method significantly outperforms existing 3D Gaussian Splatting (3D-GS) approaches, achieving state-of-the-art performance in geometric detail fidelity, temporal motion consistency, and visual realism.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper focuses on a challenging setting of simultaneously modeling geometry and appearance of hand-object interaction scenes without any object priors. We follow the trend of dynamic 3D Gaussian Splatting based methods, and address several significant challenges. To model complex hand-object interaction with mutual occlusion and edge blur, we present interaction-aware hand-object Gaussians with newly introduced optimizable parameters aiming to adopt piecewise linear hypothesis for clearer structural representation. Moreover, considering the complementarity and tightness of hand shape and object shape during interaction dynamics, we incorporate hand information into object deformation field, constructing interaction-aware dynamic fields to model flexible motions. To further address difficulties in the optimization process, we propose a progressive strategy that handles dynamic regions and static background step by step. Correspondingly, explicit regularizations are designed to stabilize the hand-object representations for smooth motion transition, physical interaction reality, and coherent lighting. Experiments show that our approach surpasses existing dynamic 3D-GS-based methods and achieves state-of-the-art performance in reconstructing dynamic hand-object interaction.

Problem

• Reconstructs hand-object interactions without prior object models
• Models mutual occlusion and motion in dynamic 4D scenes
• Handles optimization challenges in complex interaction reconstruction

Innovation

• Interaction-aware Gaussians with optimizable parameters
• Hand information integrated into object deformation field
• Progressive strategy with explicit regularization for optimization