GRASP: Granularity-Aware Region Alignment and Semantic Prototype Learning for Fine-Grained Cross-Modal Understanding in Drone Views

2026-08-10

AI Summary

This work addresses the challenges of fine-grained cross-modal understanding from drone-view imagery, specifically cross-modal focus misalignment and visual isomorphism ambiguity. To tackle these issues, the authors propose the GRASP framework, which introduces a Region-Focused Alignment (RFA) module to suppress background distractions and enable object-centric cross-modal alignment. Additionally, they design a Semantic Prototype Codebook (SPC)-based Semantic Perturbation-Enhanced Matching (SPEM) mechanism to enhance fine-grained semantic discriminability. Extensive experiments demonstrate that GRASP significantly outperforms existing methods on both the GeoText-1652 benchmark and the unseen ERA dataset, validating its effectiveness and strong generalization capability for drone-based image-text retrieval tasks.

Abstract

Fine-grained cross-modal understanding in drone views is essential for aerial vision-language navigation. However, the inherent wide field of view and overhead perspective of drone scenarios impose dual challenges on vision-language understanding. At the macro level, overwhelming background clutter in visual representations leads to Cross-Modal Focus Misalignment, where the model prioritizes global environmental similarities over specific object details. At the micro level, Visual Isomorphism creates ambiguity, where candidates share similar geometric structures yet differ only in subtle attributes. To address these challenges, we propose the Granularity-Aware Region Alignment and Semantic Prototype (GRASP) learning framework, enhancing discriminative capability through two synergistic strategies. Specifically, we introduce Region-Focused Alignment (RFA) to promote object-centric cross-modal alignment while suppressing background interference. Concurrently, to tackle visual isomorphism, we propose Semantic Perturbation Enhanced Matching (SPEM), which leverages a foreground-purified Semantic Prototype Codebook (SPC) to construct semantically perturbed negatives for fine-grained semantic discrimination. Extensive experiments on the GeoText-1652 benchmark and the unseen ERA dataset demonstrate that GRASP achieves competitive performance in drone-view fine-grained image-text retrieval, validating its effectiveness for cross-modal understanding in aerial scenarios. Our code implementation is available at https://github.com/UCAS-JC/GRASP.

Problem

• fine-grained cross-modal understanding
• drone views
• cross-modal focus misalignment
• visual isomorphism
• aerial vision-language navigation

Innovation

• Granularity-Aware Alignment
• Region-Focused Alignment
• Semantic Prototype Learning
• Visual Isomorphism
• Cross-Modal Retrieval