CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking

2026-07-16

AI Summary

This work addresses the challenge of target loss in urban drone tracking caused by occlusions by proposing a spatially aware vision-language-action model that jointly performs target localization, visibility estimation, and continuous flight action generation. The model bridges the gap from imitation learning on visible frames to spatial closed-loop control through spatial grounding continual pretraining (CPT), a three-stage curriculum-supervised fine-tuning (SFT) strategy, chain-of-thought (CoT) reasoning, and closed-loop reinforcement learning. Compared to OpenVLA, the proposed approach reduces open-loop trajectory error (ADE) by 34.1% and 35.3% on seen and unseen test scenarios, respectively, and improves closed-loop tracking success rates by 29.8% and 2.5%, substantially enhancing robustness in occlusion-prone environments.

Abstract

Dynamic target tracking is essential for Unmanned Aerial Vehicles (UAVs) operating in complex urban environments, where both the target and the camera viewpoint change continuously. Existing Vision-Language-Action (VLA) policies can track visible targets effectively, but their performance often degrades when buildings, vegetation, or roadside objects block the line of sight. During sustained occlusion, a policy may lose the target state, execute actions toward an incorrect region, and amplify this error through subsequent observations until re-acquisition becomes impossible. To this end, we present CosFly-VLA, a spatially aware VLA model that jointly grounds the target, estimates its visibility, and generates continuous flight actions through a structured prediction interface. To train this policy, we use a large-scale recipe over diverse data sources. Spatially Grounded Continued Pretraining (CPT) on a 500k mixed pool injects UAV-view depth, distance, and 3-D spatial reasoning. A three-stage Curriculum-based Supervised Fine-Tuning (SFT) process then specializes the tracker through multi-head warm-up followed by two-stage curriculum learning over natural and hard / long-occlusion data. Chain-of-Thought (CoT) training subsequently teaches recovery-oriented reasoning traces before structured answers. Finally, a closed-loop Reinforcement Learning (RL) stage optimizes tracking behavior with a multi-component reward covering stand-off tracking, grounding quality, collision avoidance, and task success. Relative to OpenVLA, CosFly-VLA-0.8B reduces open-loop Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test. Closed-loop optimization improves Success Rate (SR) by 29.8% and 2.5%, respectively. These results demonstrate progress from visible-frame imitation toward spatially grounded action-closed-loop control, evaluated under a shared oracle state history.

Problem

• UAV tracking
• occlusion
• vision-language-action
• spatial reasoning
• dynamic target tracking

Innovation

• Vision-Language-Action
• spatial grounding
• occlusion-aware tracking
• curriculum fine-tuning
• closed-loop reinforcement learning