RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videos

2026-05-16

AI Summary

This work addresses the challenging problem of jointly reconstructing the 4D dynamics of a human, unknown objects, and their interactions from monocular video, which is hindered by depth ambiguity, mutual occlusions, and motion coupling. The authors propose a three-stage framework: first, a 3D-aware foundation model stabilizes structure-from-motion to recover coarse scene geometry; second, existing human pose estimation methods are integrated to disentangle camera and object motion and align them into a common world coordinate system; third, a compositional neural radiance field combined with differentiable contact priors refines object shapes to enhance physical plausibility. This study presents the first method capable of full 4D reconstruction of human–novel-object interactions without requiring known object shapes or calibrated camera parameters, significantly outperforming existing approaches on a newly collected synchronized 4D ground-truth dataset and demonstrating the effectiveness of each component as well as the overall framework.

Abstract

Reconstructing people, objects, and their interactions in 3D is a long-standing goal for intelligent systems. Often the input is RGB video from a moving camera, making the task ill-posed; depth is ambiguous, humans and objects occlude each other, and camera and object motion entangle to create apparent motion. Most prior work addresses humans or objects in isolation, ignoring their interplay, or assumes known 3D shapes or cameras, which is impractical for real-world applications. We develop RHINO (Reconstructing Human Interactions with Novel Objects), a three-step framework that recovers in 3D a human, novel (unseen) manipulated object, and static scene in a common world frame from a monocular RGB video. First, we leverage 3D-aware foundation models to obtain cues that stabilize Structure-from-Motion (SfM) even for low-texture regions; this yields a coarse shape and apparent motion of a manipulated object from foreground pixels, and a coarse scene shape and camera motion from background pixels. Second, we estimate a human in the camera frame via an off-the-shelf method, and subtract the camera motion from apparent motion to extract the object motion; this registers the human, object, and coarse scene shapes into a common world frame. Third, we refine shapes using a compositional neural field with per-component signed-distance fields. The latter further enables differentiable contact priors that attract surfaces while penalizing interpenetration, improving the physical plausibility of the final reconstruction. For evaluation, we capture a new dataset of handheld monocular videos synchronized with a volumetric 4D capture stage, providing ground-truth shape and camera motion. RHINO outperforms state-of-the-art baselines on novel-view synthesis and 4D reconstruction. Ablations show that each stage contributes substantially. Code and data are available at https://lxxue.github.io/RHINO.

Problem

• 3D reconstruction
• human-object interaction
• monocular video
• novel objects
• motion disentanglement

Innovation

• monocular video
• 3D reconstruction
• human-object interaction
• neural signed-distance fields
• contact priors