TCMA: Text-Conditioned Multi-granularity Alignment for Drone Cross-Modal Text-Video Retrieval

2025-10-11

AI Summary

To address coarse-grained captions and semantic misalignment in drone video–text cross-modal retrieval, this paper introduces the first fine-grained, semantically rich Drone Video–Text Match Dataset (DVTMD) and proposes the Text-Conditioned Multi-level Alignment (TCMA) framework. TCMA innovatively integrates sentence-guided frame aggregation and word-level guided block alignment, incorporating a word–block filtering module and a text-adaptive dynamic temperature mechanism to achieve multi-granularity alignment—global (sentence–video), local (word–video block), and attention-aware. We establish the first unified benchmark on DVTMD and CapERA. Experimental results demonstrate state-of-the-art performance: TCMA achieves R@1 of 45.5% for text-to-video retrieval and 42.8% for video-to-text retrieval.

Technology Category

• Computer Vision: Image and Video Retrieval
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Unmanned aerial vehicles (UAVs) have become powerful platforms for real-time, high-resolution data collection, producing massive volumes of aerial videos. Efficient retrieval of relevant content from these videos is crucial for applications in urban management, emergency response, security, and disaster relief. While text-video retrieval has advanced in natural video domains, the UAV domain remains underexplored due to limitations in existing datasets, such as coarse and redundant captions. Thus, in this work, we construct the Drone Video-Text Match Dataset (DVTMD), which contains 2,864 videos and 14,320 fine-grained, semantically diverse captions. The annotations capture multiple complementary aspects, including human actions, objects, background settings, environmental conditions, and visual style, thereby enhancing text-video correspondence and reducing redundancy. Building on this dataset, we propose the Text-Conditioned Multi-granularity Alignment (TCMA) framework, which integrates global video-sentence alignment, sentence-guided frame aggregation, and word-guided patch alignment. To further refine local alignment, we design a Word and Patch Selection module that filters irrelevant content, as well as a Text-Adaptive Dynamic Temperature Mechanism that adapts attention sharpness to text type. Extensive experiments on DVTMD and CapERA establish the first complete benchmark for drone text-video retrieval. Our TCMA achieves state-of-the-art performance, including 45.5% R@1 in text-to-video and 42.8% R@1 in video-to-text retrieval, demonstrating the effectiveness of our dataset and method. The code and dataset will be released.

Problem

• Addresses drone video retrieval challenges using fine-grained text descriptions
• Develops multi-granular alignment framework for text-video matching in aerial content
• Solves dataset limitations with semantically diverse captions for UAV videos

Innovation

• Text-conditioned multi-granularity alignment framework for drone videos
• Word and patch selection module filters irrelevant visual content
• Text-adaptive dynamic temperature mechanism adjusts attention sharpness