RaceVLA: VLA-based Racing Drone Navigation with Human-like Behaviour

2025-03-04

AI Summary

To address the challenges of imitating human piloting behavior and adapting to dynamic environments in high-speed racing drone autonomous navigation, this work introduces the Vision-Language-Action (VLA) paradigm to racing scenarios for the first time. We propose a lightweight VLA architecture tailored for dynamic race tracks, along with a dedicated fine-tuning strategy that accommodates onboard dynamic visual input and simplified motion primitives. Trained on a novel multimodal racing drone dataset, our system enables language-guided action generation and real-time closed-loop control. Experiments demonstrate superior semantic generalization (45.5) and motion generalization (75.0) over OpenVLA and RT-2. In physical flight tests, the drone achieves an average speed of 1.04 m/s and a peak speed of 2.02 m/s, exhibiting robust high-speed maneuverability under complex, time-varying track conditions.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

RaceVLA presents an innovative approach for autonomous racing drone navigation by leveraging Visual-Language-Action (VLA) to emulate human-like behavior. This research explores the integration of advanced algorithms that enable drones to adapt their navigation strategies based on real-time environmental feedback, mimicking the decision-making processes of human pilots. The model, fine-tuned on a collected racing drone dataset, demonstrates strong generalization despite the complexity of drone racing environments. RaceVLA outperforms OpenVLA in motion (75.0 vs 60.0) and semantic generalization (45.5 vs 36.3), benefiting from the dynamic camera and simplified motion tasks. However, visual (79.6 vs 87.0) and physical (50.0 vs 76.7) generalization were slightly reduced due to the challenges of maneuvering in dynamic environments with varying object sizes. RaceVLA also outperforms RT-2 across all axes - visual (79.6 vs 52.0), motion (75.0 vs 55.0), physical (50.0 vs 26.7), and semantic (45.5 vs 38.8), demonstrating its robustness for real-time adjustments in complex environments. Experiments revealed an average velocity of 1.04 m/s, with a maximum speed of 2.02 m/s, and consistent maneuverability, demonstrating RaceVLA's ability to handle high-speed scenarios effectively. These findings highlight the potential of RaceVLA for high-performance navigation in competitive racing contexts. The RaceVLA codebase, pretrained weights, and dataset are available at this http URL: https://racevla.github.io/

Problem

• Autonomous drone navigation with human-like behavior.
• Real-time environmental feedback for adaptive navigation.
• High-speed maneuverability in complex racing environments.

Innovation

• VLA-based navigation mimics human-like behavior
• Real-time environmental feedback adapts drone strategies
• High-speed maneuverability in complex racing environments