UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery

2026-07-09

AI Summary

This work addresses the limitations of existing drone-based visual grounding methods, which are confined to text-only queries and single-target outputs, rendering them inadequate for multimodal instructions or scenarios involving zero or multiple targets. To overcome this, the authors propose a unified referring framework and introduce UniRef-UAV, the first drone benchmark supporting textual, visual, and multimodal queries. They further design UAV-URNet, a detection-based baseline model that leverages a shared query-space mapping and set prediction mechanism to enable end-to-end prediction of a variable number of targets. This study is the first to jointly expand both query modalities and output cardinality in drone scenarios, unifying zero-shot, single-target, and multi-target grounding within a single framework, and establishes both in-domain and cross-domain evaluation protocols. Experiments demonstrate that UAV-URNet is lightweight, efficient, consistently accurate in identifying absence-of-target cases, and benefits significantly from multimodal queries that reduce visual ambiguity and enhance semantic alignment.

Abstract

Unmanned aerial vehicles (UAVs) increasingly rely on visual grounding capabilities to localize task-relevant targets from diverse instructions in complex aerial scenes. Existing referring expression comprehension (REC) benchmarks and methods, however, are largely built around text-only queries and single-object outputs, which limits their applicability to practical UAV scenarios involving reference images, multimodal instructions, absent targets, and multiple valid target instances. To address this gap, we introduce \emph{Universal Referring}, a generalized UAV referring task that jointly expands the query modality and the output cardinality. We construct \emph{UniRef-UAV}, a multimodal benchmark that supports text-only, image-only, and text+image queries with modality-dependent target cardinality, where text-only and text+image queries admit no-target, single-target, and multi-target grounding while image-only queries focus on existence-aware single-instance grounding. It also provides in-domain and cross-domain evaluation protocols for visual-query generalization. We further present \emph{UAV-URNet}, a detection-style baseline that maps heterogeneous queries into a shared query space and predicts variable-size target sets through set prediction. Extensive experiments show that UAV-URNet provides a stable and reproducible baseline with more consistent no-target discrimination and a more lightweight, reproducible implementation than large general-purpose MLLMs. Additional domain analysis, query-representation analysis, and ablation studies demonstrate that multimodal queries help reduce visual-query ambiguity and promote a more unified query--target alignment space. The annotations, visual query crops/images, train/validation/test splits, evaluation scripts, and baseline code will be made publicly available to facilitate reproducible research.

Problem

• referring expression comprehension
• UAV imagery
• multimodal queries
• visual grounding
• universal referring

Innovation

• Universal Referring
• Multimodal Benchmark
• UAV Imagery
• Set Prediction
• Visual Grounding