JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

2026-07-02

AI Summary

This work addresses the common issue of physically implausible artifacts—such as object floating or interpenetration—in text-driven hand–object interaction generation, which often arises due to insufficient physical constraints. The authors propose the first single-stage diffusion framework that jointly generates 3D hand–object motion sequences and dynamic distance field contact maps. By modeling contact as an intrinsic modality, the method learns its coupling with motion during training and leverages contact-aware guidance during inference to enhance geometric consistency. This enables end-to-end co-modeling of contact and motion, significantly improving temporal stability and physical plausibility. Experiments on the GRAB and ARCTIC datasets demonstrate that the proposed approach outperforms existing methods in both textual alignment and physical realism, effectively mitigating penetration and floating artifacts.

Abstract

Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.

Problem

• Hand Object Interaction
• Contact Modeling
• Physical Plausibility
• Text-driven Generation
• Temporal Contact

Innovation

• joint generation
• contact maps
• diffusion model
• hand-object interaction
• physical plausibility