FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking

2026-06-29

AI Summary

This work addresses the challenge of visual aerial tracking in GPS-denied environments, where existing methods rely heavily on labor-intensive annotations and struggle to achieve efficient, reliable perception. The authors propose a unified perception-and-tracking framework that leverages an editable Gaussian splatting simulator to automatically generate photorealistic images with 6-DoF ground-truth labels. Integrating a multi-head neural perception module with a physics-aware tracking mechanism, the approach achieves, for the first time, explicit separation of target Gaussians from background clutter and enables zero-shot sim-to-real transfer. Key innovations include a fully automated annotation pipeline, reprojection consistency constraints, and an extended Kalman filter augmented with class-conditional dynamic priors. Evaluated in real-world scenarios, the method attains 96–100% category-level accuracy, runs onboard at 25 Hz, and achieves 100% success rates in both F1-tenth and doorframe tracking tasks, substantially outperforming current baselines.

Abstract

Vision-based aerial tracking is critical in GPS-denied environments. Reliable perception for tracking depends on large-scale labeled data, yet most photorealistic datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconTrack, a unified perception-and-tracking framework that (i) leverages a photorealistic editable simulator for automated label generation and (ii) combines multi-head perception with physics-aware tracking for zero-shot sim-to-real transfer. FalconTrack provides an automated labeling pipeline in a Gaussian Splatting simulator that isolates target Gaussians from short object videos and composites them with randomized backgrounds to generate RGB, mask, class, and 6-DoF pose labels, producing about 10k labeled images in under 20 minutes. Using this dataset, we train a multi-head perception module with staged learning and reprojection consistency, and fuse its outputs with class-conditioned dynamics priors in an EKF for tracking. Our perception model outperforms two baselines and reaches 96-100% class accuracy in zero-shot sim-to-real transfer on three geometrically diverse objects and two environments, while maintaining consistent performance in unseen simulated and real scenes. In real hardware closed-loop visual tracking, the onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments, while a mask-centered vision baseline drops to 60% success on F1-tenth during fast out-of-view scenarios.

Problem

• vision-based aerial tracking
• photorealistic dataset
• automated labeling
• sim-to-real transfer
• GPS-denied environments

Innovation

• photorealistic simulation
• automated labeling
• physics-aware tracking
• zero-shot sim-to-real
• Gaussian Splatting