AgenticFocus: Object-Preserving Mixed Reality Synthesis from Human FPV Video for Dexterous Humanoid Learning

2026-07-09

AI Summary

This work addresses the limitations of existing methods for training humanoid robot policies from first-person videos, which often suffer from hand-object occlusions, oversimplified motions, or reliance on specialized hardware. The authors propose a hardware-free mixed-reality synthesis pipeline that recovers occluded object geometry, reconstructs complete hand motion, and leverages relative camera alignment with hierarchical compositing to achieve high-fidelity, object-preserving motion retargeting. This approach is the first to effectively mitigate hand-object occlusion under ordinary video conditions, substantially enhancing motion naturalness. Experimental results demonstrate that the generated data outperforms baseline methods in trajectory accuracy and wrist smoothness, achieving a SPARC score of −5.18—significantly better than competing models (−5.56 and −6.05).

Abstract

Human egocentric video is a scalable supervision source for humanoid policy learning, but current pipelines struggle with hand-object occlusion, oversimplified motion, or specialized capture hardware. We introduce AgenticFocus, a Mixed Reality synthesis pipeline that converts ordinary first-person-view human videos into robot-trainable demonstrations by restoring occluded object geometry, reconstructing full-hand motion, and retargeting it to a humanoid embodiment through camera-relative alignment and layered compositing. The resulting dataset pairs focused visual observations with synchronized robot actions and states. AgenticFocus achieves lower trajectory error and smoother wrist motion than cross-embodiment baselines, with SPARC scores of -5.18 versus -5.56 and -6.05.

Problem

• humanoid learning
• hand-object occlusion
• first-person-view video
• motion reconstruction
• mixed reality synthesis

Innovation

• Mixed Reality Synthesis
• Object-Preserving Reconstruction
• First-Person View (FPV)
• Humanoid Policy Learning
• Hand-Object Occlusion Handling