RS-vHeat: Heat Conduction Guided Efficient Remote Sensing Foundation Model

2024-11-27

arXiv.org

AI Summary

To address the low computational efficiency and poor interpretability of foundation models in high-resolution remote sensing image modeling, this paper proposes a novel heat-conduction-guided representation learning paradigm. We introduce the Heat Conduction Operator (HCO), the first of its kind, which enables parallel local correlation propagation and global receptive field modeling at O(N^1.5) complexity. Coupled with frequency-domain hierarchical masking and multi-domain reconstruction-based self-supervision, our approach enhances representation robustness. Furthermore, we integrate multimodal features to construct a lightweight yet highly efficient model. Extensive evaluation across four downstream tasks and ten remote sensing benchmarks demonstrates consistent state-of-the-art performance: 84% reduction in memory footprint, 24% fewer FLOPs, and a 2.7× throughput improvement—achieving superior accuracy, efficiency, and physics-informed interpretability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Remote sensing foundation models largely break away from the traditional paradigm of designing task-specific models, offering greater scalability across multiple tasks. However, they face challenges such as low computational efficiency and limited interpretability, especially when dealing with large-scale remote sensing images. To overcome these, we draw inspiration from heat conduction, a physical process modeling local heat diffusion. Building on this idea, we are the first to explore the potential of using the parallel computing model of heat conduction to simulate the local region correlations in high-resolution remote sensing images, and introduce RS-vHeat, an efficient multi-modal remote sensing foundation model. Specifically, RS-vHeat 1) applies the Heat Conduction Operator (HCO) with a complexity of $O(N^{1.5})$ and a global receptive field, reducing computational overhead while capturing remote sensing object structure information to guide heat diffusion; 2) learns the frequency distribution representations of various scenes through a self-supervised strategy based on frequency domain hierarchical masking and multi-domain reconstruction; 3) significantly improves efficiency and performance over state-of-the-art techniques across 4 tasks and 10 datasets. Compared to attention-based remote sensing foundation models, we reduce memory usage by 84%, FLOPs by 24% and improves throughput by 2.7 times. The code will be made publicly available.

Problem

• Improves computational efficiency in remote sensing models
• Enhances interpretability of large-scale remote sensing images
• Reduces memory usage and FLOPs in foundation models

Innovation

• Heat Conduction Operator reduces computational overhead
• Self-supervised learning via frequency domain masking
• Improves efficiency and performance across multiple tasks