HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

2026-06-26

AI Summary

Existing monocular 4D reconstruction methods struggle to handle severe occlusions and complex dynamics in multi-object interactions, often failing to produce physically plausible and temporally consistent 4D reconstructions. This work proposes the first agent-based framework for monocular 4D reconstruction, integrating vision-language models, multi-level human feedback, 3D generation, and 4D propagation techniques to effectively mitigate depth ambiguity and occlusion challenges. The study introduces MVOIK-4D, the first open-world benchmark for monocular 4D interactive reconstruction, along with a multidimensional evaluation protocol. The proposed method achieves state-of-the-art performance across most metrics and demonstrates that minimal human feedback significantly enhances reconstruction quality and effectively boosts fine-tuning performance on downstream tasks.

Abstract

Extracting dynamic 4D object interactions from massive, in-the-wild monocular videos offers a highly efficient data collection pathway for scaling Embodied AI and training VLAs. However, existing monocular 4D reconstruction methods primarily focus on isolated objects, often failing under the severe occlusions and complex dynamics inherent in multi-object interactions. To bridge this gap, we propose HAT-4D, the first agentic framework designed to reconstruct the 3D geometry, temporal dynamics, and physical interactions of multiple objects from a single video. By integrating VLMs with a multi-level human-in-the-loop feedback mechanism, HAT-4D efficiently resolves depth ambiguities and interaction-induced occlusions during 3D generation and 4D propagation, yielding physically plausible assets without relying on expensive multicamera rigs. As a scalable data engine, HAT-4D facilitates the creation of MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction, accompanied by a novel multi-dimensional evaluation protocol focused on physical plausibility and temporal consistency. Extensive experiments demonstrate that HAT-4D achieves SOTA performance on most evaluation metrics, while maintaining competitive semantic alignment. Ablation studies show that introducing a small amount of human feedback improves interaction reconstruction. Moreover, the data produced by HAT-4D effectively improves baseline performance when used for fine-tuning. Our data and code are available at https://lijiaxin0111.github.io/HAT4D/

Problem

• monocular video
• 4D reconstruction
• multi-object interactions
• occlusion
• dynamic scenes

Innovation

• 4D reconstruction
• monocular video
• human-agent collaboration
• multi-object interaction
• visual language models