DyTact: Capturing Dynamic Contacts in Hand-Object Manipulation

2025-06-03

AI Summary

Dynamic hand-object contact reconstruction is critical for AI-driven animation, XR, and robotic manipulation, yet remains challenging due to severe occlusions, complex geometries, and limitations of existing methods—including low accuracy and high computational overhead. This paper introduces the first dynamic articulated representation based on 2D Gaussian surfels, integrating the MANO hand prior with differentiable rendering. We propose a contact-guided adaptive sampling strategy and a temporal-aware high-frequency deformation optimization module to enable density-aware contact modeling and enforce temporal consistency. Our method achieves high-fidelity contact estimation under markerless conditions, attaining state-of-the-art accuracy (Contact F1 +3.2%), improved novel-view synthesis (PSNR +2.1 dB), 2.3× faster optimization, and 37% reduced GPU memory consumption.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Manipulation
• Humans and AI: Interaction Techniques and Devices

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Reconstructing dynamic hand-object contacts is essential for realistic manipulation in AI character animation, XR, and robotics, yet it remains challenging due to heavy occlusions, complex surface details, and limitations in existing capture techniques. In this paper, we introduce DyTact, a markerless capture method for accurately capturing dynamic contact in hand-object manipulations in a non-intrusive manner. Our approach leverages a dynamic, articulated representation based on 2D Gaussian surfels to model complex manipulations. By binding these surfels to MANO meshes, DyTact harnesses the inductive bias of template models to stabilize and accelerate optimization. A refinement module addresses time-dependent high-frequency deformations, while a contact-guided adaptive sampling strategy selectively increases surfel density in contact regions to handle heavy occlusion. Extensive experiments demonstrate that DyTact not only achieves state-of-the-art dynamic contact estimation accuracy but also significantly improves novel view synthesis quality, all while operating with fast optimization and efficient memory usage. Project Page: https://oliver-cong02.github.io/DyTact.github.io/ .

Problem

• Reconstructing dynamic hand-object contacts for realistic AI and robotics
• Overcoming occlusions and surface details in contact capture
• Improving accuracy and efficiency in dynamic contact estimation

Innovation

• Markerless capture for dynamic hand-object contacts
• 2D Gaussian surfels with MANO mesh binding
• Contact-guided adaptive sampling for occlusion handling