CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction

2026-05-20

AI Summary

Reconstructing 4D representations of hand–object interactions—including hand pose, object shape, 6D object pose, and contact state—from open-world monocular videos is challenged by occlusions, unknown geometry, and hand–object misalignment. This work proposes CHOIR, a novel framework that explicitly models contact as a dynamically updated constraint. Starting from a coarse interaction sequence initialized with visual priors, CHOIR integrates generative spatial refinement, ray-based depth correction, and contact correspondence estimation, followed by a contact-aware joint optimization to enforce geometric, temporal, and physical consistency. Evaluated on both in-the-wild and controlled videos, the method significantly outperforms existing approaches, achieving notable improvements in object reconstruction quality, physical plausibility, and temporal coherence.

Abstract

We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present CHOIR, a Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods.

Problem

• hand-object interaction
• 4D reconstruction
• monocular video
• contact modeling
• object pose estimation

Innovation

• contact-aware
• 4D hand-object interaction
• monocular reconstruction
• spatial rectification
• joint optimization