AdaptFly: Prompt-Guided Adaptation of Foundation Models for Low-Altitude UAV Networks

2025-11-12

AI Summary

Semantic segmentation models for low-altitude unmanned aerial vehicle (UAV) networks suffer severe performance degradation under varying weather conditions, illumination changes, and viewing angles. Method: We propose a weight-free, prompt-guided test-time adaptation (TTA) framework. It features a dual-mode collaborative mechanism: (i) lightweight token-based prompt retrieval for rapid adaptation, and (ii) gradient-free sparse visual prompt optimization for enhanced robustness. Prompt search is performed via the covariance matrix adaptation evolution strategy (CMA-ES), while activation-statistics-driven adaptive triggering and a global memory bank enable cross-UAV knowledge sharing. Contribution/Results: The framework is hardware-agnostic—supporting both resource-constrained and resource-rich UAVs—and drastically reduces communication overhead. Extensive experiments on UAVid, VDD, and real-world multi-weather datasets demonstrate substantial improvements in segmentation accuracy and system robustness over static models and state-of-the-art TTA methods.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Low-altitude Unmanned Aerial Vehicle (UAV) networks rely on robust semantic segmentation as a foundational enabler for distributed sensing-communication-control co-design across heterogeneous agents within the network. However, segmentation foundation models deteriorate quickly under weather, lighting, and viewpoint drift. Resource-limited UAVs cannot run gradient-based test-time adaptation, while resource-massive UAVs adapt independently, wasting shared experience. To address these challenges, we propose AdaptFly, a prompt-guided test-time adaptation framework that adjusts segmentation models without weight updates. AdaptFly features two complementary adaptation modes. For resource-limited UAVs, it employs lightweight token-prompt retrieval from a shared global memory. For resource-massive UAVs, it uses gradient-free sparse visual prompt optimization via Covariance Matrix Adaptation Evolution Strategy. An activation-statistic detector triggers adaptation, while cross-UAV knowledge pool consolidates prompt knowledge and enables fleet-wide collaboration with negligible bandwidth overhead. Extensive experiments on UAVid and VDD benchmarks, along with real-world UAV deployments under diverse weather conditions, demonstrate that AdaptFly significantly improves segmentation accuracy and robustness over static models and state-of-the-art TTA baselines. The results highlight a practical path to resilient, communication-efficient perception in the emerging low-altitude economy.

Problem

• Addresses semantic segmentation degradation in UAV networks under environmental changes
• Enables lightweight adaptation for resource-limited drones without weight updates
• Facilitates fleet-wide knowledge sharing to improve segmentation accuracy collaboratively

Innovation

• Prompt-guided adaptation without weight updates
• Lightweight token-prompt retrieval for limited UAVs
• Gradient-free visual prompt optimization via CMA-ES