ComPose: When to Trust Hands for Object Pose Tracking

2026-05-22

AI Summary

This work addresses the significant performance degradation of existing 6DoF object pose tracking methods under severe hand occlusion, which often rely on strong priors such as depth data or 3D templates. The proposed ComPose framework uniquely treats hand motion not as a source of occlusion but as a complementary cue for pose estimation. By leveraging foundation models to extract hand-object features, ComPose adaptively selects informative keypoints within a unified pipeline, fuses geometric visibility constraints with learning-driven pose refinement, and explicitly models temporal consistency between rotation and translation. This enables robust, temporally coherent tracking without requiring post-hoc smoothing. The method achieves substantial gains in accuracy and stability under heavy occlusion and geometric ambiguity, and successfully generalizes to robotic manipulation tasks, enabling reconstruction of hand–object interaction trajectories from monocular RGB videos.

Abstract

Reconstructing the motion of objects from videos is a key component for embodied AI and robot manipulation. While diverse approaches to object pose tracking have been studied, they rely heavily on strong external priors, such as depth data or 3D templates, and remain highly vulnerable to severe occlusions by hand grasps despite the use of explicit masks. In this work, we present ComPose, a 6DoF object tracking framework designed for hand-aware object pose estimation from RGB video. Rather than treating the hand purely as an occluder, our method harmonizes hand motions as a \textit{complementary cue} for object tracking. In detail, we recover a variety of object motions over time by combining object and hand cues from foundation models within a unified tracking pipeline. Here, ComPose adaptively selects informative hand joints, combines object- and hand-derived cues for motion estimation, and refines the resulting object motion using visible geometric evidence and a learned correction. We further enforce the temporal consistency over both rotation and translation, yielding stable 3D object trajectories over time without any external smoothing. Extensive experiments show that our method is accurate, efficient, and robust under severe hand occlusion and geometric ambiguity. In addition, the resulting trajectories can also effectively transfer to downstream robot manipulation by enabling robots to reconstruct human actions from online videos.

Problem

• object pose tracking
• hand occlusion
• 6DoF
• RGB video
• embodied AI

Innovation

• hand-aware pose estimation
• 6DoF object tracking
• complementary cue
• foundation models
• temporal consistency