EgoTrack3D: A Modular Framework for Egocentric 3D Object Tracking

2026-08-08

AI Summary

This work addresses the challenge of dynamic 3D scene modeling from egocentric videos, where drastic viewpoint changes and partial occlusions hinder consistent long-term tracking of both static and dynamic objects. To overcome this, the authors propose a modular framework that constructs globally consistent 3D object representations from monocular RGB video by integrating 2D-to-3D lifting, point-based motion scoring, and voxel-merging heuristics. A key innovation is an interaction-guided dynamic association mechanism that ensures robustness under sparse or noisy observations. The method achieves, for the first time in a general setting, continuous 3D tracking of both static and dynamic objects, outperforming the strongest baseline by 11% in PCL metric on the Aria Digital Twin dataset, thereby demonstrating its effectiveness under real-world deployment conditions.

Abstract

Understanding 3D scenes from egocentric video is fundamental for robotics and autonomous navigation, yet rapid viewpoint changes and partial occlusions make building structured representations challenging. Existing 3D tracking and scene graph construction methods primarily address explicit interactions or assume static scenes, limiting their ability to capture complex dynamics. We introduce EgoTrack3D, a modular framework that reconstructs and maintains a dynamic 3D scene representation directly from egocentric RGB video. The framework lifts 2D segmentation masks into a global 3D coordinate frame, using a point-based motion scoring mechanism alongside a voxel-based merging heuristic to associate object tracks. EgoTrack3D maintains accurate representations over time, achieving an 11% improvement in percentage of correct locations (PCL) relative to the strongest baseline on the Aria Digital Twin (ADT) dataset, while addressing the more general setting of persistent 3D tracking for both static and dynamic objects. Furthermore, to demonstrate the system's robustness under degraded conditions that simulate real-world deployment constraints, we replace dense depth maps with sparse 3D bounding box estimation and integrate interaction-guided dynamic association, enabling EgoTrack3D to maintain accurate spatial representations despite noisy observations.

Problem

• egocentric vision
• 3D object tracking
• dynamic scene representation
• partial occlusion
• viewpoint change

Innovation

• egocentric vision
• 3D object tracking
• dynamic scene representation
• modular framework
• sparse depth estimation