WeatherPrompt: Multi-modality Representation Learning for All-Weather Drone Visual Geo-Localization

2025-08-13

AI Summary

Drone-based visual geo-localization suffers significant performance degradation under adverse weather conditions such as rain and fog. Existing approaches are limited by poor generalization across predefined weather categories and ineffective disentanglement of scene- and weather-specific features. To address these challenges, we propose a multimodal weather-invariant representation learning framework. First, we leverage a frozen multimodal large language model to generate zero-shot textual weather descriptors—including type and intensity—without additional training. Second, we introduce a text-driven dynamic gating mechanism to explicitly decouple scene and weather representations. Finally, we jointly optimize the model via image–text contrastive learning and cross-modal matching objectives. Experiments demonstrate substantial robustness improvements under unseen adverse conditions—including nighttime, fog, and snow—with Recall@1 gains of 13.37% and 18.69%, respectively, significantly outperforming state-of-the-art methods.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Visual geo-localization for drones faces critical degradation under weather perturbations, eg, rain and fog, where existing methods struggle with two inherent limitations: 1) Heavy reliance on limited weather categories that constrain generalization, and 2) Suboptimal disentanglement of entangled scene-weather features through pseudo weather categories. We present WeatherPrompt, a multi-modality learning paradigm that establishes weather-invariant representations through fusing the image embedding with the text context. Our framework introduces two key contributions: First, a Training-free Weather Reasoning mechanism that employs off-the-shelf large multi-modality models to synthesize multi-weather textual descriptions through human-like reasoning. It improves the scalability to unseen or complex weather, and could reflect different weather strength. Second, to better disentangle the scene and weather feature, we propose a multi-modality framework with the dynamic gating mechanism driven by the text embedding to adaptively reweight and fuse visual features across modalities. The framework is further optimized by the cross-modal objectives, including image-text contrastive learning and image-text matching, which maps the same scene with different weather conditions closer in the respresentation space. Extensive experiments validate that, under diverse weather conditions, our method achieves competitive recall rates compared to state-of-the-art drone geo-localization methods. Notably, it improves Recall@1 by +13.37% under night conditions and by 18.69% under fog and snow conditions.

Problem

• Improves drone visual geo-localization under diverse weather conditions
• Enhances generalization to unseen or complex weather scenarios
• Disentangles scene and weather features for robust representations

Innovation

• Training-free Weather Reasoning with multi-modality models
• Dynamic gating mechanism for feature disentanglement
• Cross-modal objectives for representation optimization