Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representation

2025-12-01

AI Summary

Existing hand-object interaction video generation methods struggle to simultaneously achieve scalability and interaction fidelity due to inherent limitations of 2D or 3D representations. To address this, we propose a novel structure- and contact-aware representation that requires no 3D annotations, explicitly modeling contact states, occlusion relationships, and geometric structural constraints. Our approach adopts a shared-specialized joint generation paradigm, integrating spatiotemporal consistency modeling under 2D supervision to precisely capture complex physical interactions and enable generalization to open-world scenes. Evaluated on two real-world datasets, our method significantly outperforms state-of-the-art approaches, generating physically plausible, temporally coherent, high-fidelity interaction videos. It achieves substantial improvements across three key dimensions: interaction fidelity (e.g., accurate contact localization and force alignment), dynamic coherence (e.g., smooth motion transitions and consistent object dynamics), and cross-scene generalization (e.g., robust performance under unseen object geometries and hand poses).

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Low Level & Physics-based Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Generating realistic hand-object interactions (HOI) videos is a significant challenge due to the difficulty of modeling physical constraints (e.g., contact and occlusion between hands and manipulated objects). Current methods utilize HOI representation as an auxiliary generative objective to guide video synthesis. However, there is a dilemma between 2D and 3D representations that cannot simultaneously guarantee scalability and interaction fidelity. To address this limitation, we propose a structure and contact-aware representation that captures hand-object contact, hand-object occlusion, and holistic structure context without 3D annotations. This interaction-oriented and scalable supervision signal enables the model to learn fine-grained interaction physics and generalize to open-world scenarios. To fully exploit the proposed representation, we introduce a joint-generation paradigm with a share-and-specialization strategy that generates interaction-oriented representations and videos. Extensive experiments demonstrate that our method outperforms state-of-the-art methods on two real-world datasets in generating physics-realistic and temporally coherent HOI videos. Furthermore, our approach exhibits strong generalization to challenging open-world scenarios, highlighting the benefit of our scalable design. Our project page is https://hgzn258.github.io/SCAR/.

Problem

• Generating realistic hand-object interaction videos
• Addressing scalability and fidelity trade-off in HOI representation
• Enabling generalization to open-world scenarios without 3D annotations

Innovation

• Structure and contact-aware representation without 3D annotations
• Joint-generation paradigm with share-and-specialization strategy
• Scalable supervision for open-world generalization