UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models

2026-04-02

AI Summary

This work addresses the challenges of semantic understanding and real-time control in embodied visual tracking for drones operating in dynamic urban environments. To this end, the authors introduce a large-scale multimodal tracking dataset and benchmark comprising 890,000 frames across 176 tasks. They propose an efficient Vision-Language-Action tracking model based on the π₀.₅ architecture, featuring a temporal compression network to reduce redundancy, a dual-branch decoder for cross-modal feature disentanglement, and a parallel structure combining a spatial-aware localization head with a flow-matching action expert to enhance geometric prior modeling and zero-shot generalization. Evaluated on the CARLA simulation platform, the model achieves a success rate of 61.76% on long-range pedestrian tracking tasks, maintains tracking for an average of 269.65 frames, and attains an inference latency of only 0.0571 seconds—33.4% faster than the baseline.

Abstract

Embodied visual tracking is crucial for Unmanned Aerial Vehicles (UAVs) executing complex real-world tasks. In dynamic urban scenarios with complex semantic requirements, Vision-Language-Action (VLA) models show great promise due to their cross-modal fusion and continuous action generation capabilities. To benchmark multimodal tracking in such environments, we construct a dedicated evaluation benchmark and a large-scale dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects. Furthermore, to address temporal feature redundancy and the lack of spatial geometric priors in existing VLA models, we propose an improved VLA tracking model, UAV-Track VLA. Built upon the $π_{0.5}$ architecture, our model introduces a temporal compression net to efficiently capture inter-frame dynamics. Additionally, a parallel dual-branch decoder comprising a spatial-aware auxiliary grounding head and a flow matching action expert is designed to decouple cross-modal features and generate fine-grained continuous actions. Systematic experiments in the CARLA simulator validate the superior end-to-end performance of our method. Notably, in challenging long-distance pedestrian tracking tasks, UAV-Track VLA achieves a 61.76\% success rate and 269.65 average tracking frames, significantly outperforming existing baselines. Furthermore, it demonstrates robust zero-shot generalization in unseen environments and reduces single-step inference latency by 33.4\% (to 0.0571s) compared to the original $π_{0.5}$, enabling highly efficient, real-time UAV control. Data samples and demonstration videos are available at: https://github.com/Hub-Tian/UAV-Track\_VLA.

Problem

• Embodied visual tracking
• Unmanned Aerial Vehicles
• Vision-Language-Action models
• Multimodal tracking
• Temporal feature redundancy

Innovation

• Vision-Language-Action (VLA)
• temporal compression
• spatial-aware grounding
• embodied aerial tracking
• zero-shot generalization