VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision

2026-06-16

AI Summary

This work addresses the challenge of learning vision-language-action (VLA) policies for obstacle avoidance from unlabeled first-person navigation videos, where trajectories annotated with obstacle awareness in the robot’s coordinate frame are unavailable. The proposed method reconstructs local scene geometry from monocular video, samples goals in the form of text, images, or spatial waypoints, and generates geometry-guided collision-free trajectories. A flow-matching framework is employed to train the VLA policy by distilling geometric planning capabilities into a purely visual policy, eliminating the need for runtime geometric inputs. This approach enables, for the first time, video-driven VLA training from in-the-wild footage without explicit geometric supervision. The authors also introduce VEGA-Bench, a large-scale evaluation benchmark. Experiments demonstrate a 33.0% reduction in collision rate and a 17.9% increase in obstacle clearance on VEGA-Bench; real-world evaluations show over 150% higher task success, 66.7% fewer collisions, and 60.0% greater obstacle clearance.

Abstract

We introduce VEGA, an approach for training navigation VisionLanguage-Action (VLA) models from unlabeled egocentric navigation videos. Internet-scale egocentric videos provide a scalable source of navigation-relevant visual observations, capturing cluttered scenes, close-range obstacles, and natural human motion through real-world spaces. However, these videos are not directly usable for policy learning because they do not provide obstacle-aware trajectories conditioned on explicit navigation goals in the robot's coordinate frame. VEGA addresses this gap by reconstructing local scene geometry from monocular video, sampling navigation goals (represented as text, image, or spatial waypoints) and generating obstacle-aware trajectories using the constructed geometry. The resulting trajectory distribution is then used to train a flow-matching VLA navigation policy. By using geometry exclusively during training, VEGA distills obstacle-aware planning directly into a vision-based policy. Furthermore, we introduce VEGA-Bench, a benchmark containing 250k scenes and approximately 5 million navigation goals paired with scene geometry, designed to evaluate goal progress, collision avoidance, and obstacle clearance of VLAs. Our evaluation shows that VEGA achieves competitive goal progress while reducing collisions by 33.0% and improving obstacle clearance by 17.9% over the strongest baseline on VEGABench, while improving success by at least 150.0%, reducing collisions by at least 66.7%, and improving obstacle clearance by at least 60.0% in real-world trials. Ultimately, we demonstrate that video-derived geometric supervision provides a scalable and effective signal for training obstacle-aware navigation VLAs. The code and benchmark will be released at the time of publication.

Problem

• egocentric video
• navigation
• obstacle-aware trajectory
• Vision-Language-Action
• geometric supervision

Innovation

• Vision-Language-Action (VLA)
• egocentric video
• geometric trajectory supervision
• flow-matching policy
• obstacle-aware navigation