JitTrack: Onboard Multi-Object Tracking Against Viewpoint Jitter for Agile UAVs

2026-08-11

AI Summary

This work addresses the challenge of achieving stable and accurate multi-object tracking for agile drones under severe viewpoint jitter, a scenario where existing methods suffer from inadequate modeling of camera ego-motion and neglect of real-world deployment constraints. To this end, we propose JitTrack, a novel framework that explicitly incorporates viewpoint jitter into a query-based Transformer tracker for the first time. Our approach introduces a motion-aware query correction mechanism and a motion-informed denoising training strategy, embedded within a closed-loop perception-planning-control system. Evaluated on public UAV MOT benchmarks, JitTrack significantly outperforms current baselines, and real-world flight experiments demonstrate its robustness in highly dynamic environments, enabling collision-free, physically feasible active multi-object tracking even under intense camera motion.

Abstract

Multi-object tracking (MOT) onboard agile unmanned aerial vehicles (UAVs) remains challenging due to severe viewpoint jitter induced by camera ego-motion. Rapid attitude changes during flight often lead to significant target displacement across frames, causing inaccurate target association and degraded tracking performance. Existing UAV MOT methods are primarily evaluated on offline benchmarks and seldom address the practical requirements of real-world onboard deployment, including robustness to camera motion and active target following. To address these challenges, we propose JitTrack, an active onboard multi-object tracking framework that accommodates drone dynamics and camera ego-motion. Built upon a query-based transformer tracker, JitTrack introduces semantic refinement to improve the detection of emerging targets, motion-aware query rectification to compensate for target misalignment caused by viewpoint jitter, and a motion-inspired denoising training strategy that simulates camera motion patterns for robust supervision. Furthermore, we develop a perception-planning-control closed-loop tracking pipeline for real-world deployment, enabling collision-free and physically feasible target following on agile UAVs. Extensive experiments on public UAV MOT benchmarks demonstrate consistent improvements over the baseline method, while real-world flight experiments validate the effectiveness and practicality of JitTrack for robust onboard visual tracking under viewpoint jitter.

Problem

• multi-object tracking
• viewpoint jitter
• UAV
• camera ego-motion
• onboard tracking

Innovation

• multi-object tracking
• viewpoint jitter
• onboard UAV
• motion-aware query rectification
• closed-loop tracking