RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretation

2025-04-04

AI Summary

Remote sensing image interpretation has long been constrained by unimodal modeling, limiting effective fusion of complementary modalities—such as optical, SAR, and multispectral data—to mitigate ambiguity and uncertainty. To address this, we introduce the first general-purpose multimodal foundation model for remote sensing, with 14.7 billion parameters. Our method features a novel hierarchical Mixture-of-Experts (MoE) architecture, integrating physics-informed self-supervised pretraining, sensor-specific radiometric modeling, dynamic sparse activation, and joint contrastive–masked learning across heterogeneous sources. The model natively supports six fundamental tasks—including classification, detection, and segmentation—and establishes new state-of-the-art results on 23 benchmarks. Furthermore, it enables dynamic pruning: compressed to 1.0 billion parameters, it retains competitive performance. Deployed in real-world applications—including emergency response, marine monitoring, and urban planning—the model demonstrates strong practical utility and scalability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The rapid advancement of foundation models has revolutionized visual representation learning in a self-supervised manner. However, their application in remote sensing (RS) remains constrained by a fundamental gap: existing models predominantly handle single or limited modalities, overlooking the inherently multi-modal nature of RS observations. Optical, synthetic aperture radar (SAR), and multi-spectral data offer complementary insights that significantly reduce the inherent ambiguity and uncertainty in single-source analysis. To bridge this gap, we introduce RingMoE, a unified multi-modal RS foundation model with 14.7 billion parameters, pre-trained on 400 million multi-modal RS images from nine satellites. RingMoE incorporates three key innovations: (1) A hierarchical Mixture-of-Experts (MoE) architecture comprising modal-specialized, collaborative, and shared experts, effectively modeling intra-modal knowledge while capturing cross-modal dependencies to mitigate conflicts between modal representations; (2) Physics-informed self-supervised learning, explicitly embedding sensor-specific radiometric characteristics into the pre-training objectives; (3) Dynamic expert pruning, enabling adaptive model compression from 14.7B to 1B parameters while maintaining performance, facilitating efficient deployment in Earth observation applications. Evaluated across 23 benchmarks spanning six key RS tasks (i.e., classification, detection, segmentation, tracking, change detection, and depth estimation), RingMoE outperforms existing foundation models and sets new SOTAs, demonstrating remarkable adaptability from single-modal to multi-modal scenarios. Beyond theoretical progress, it has been deployed and trialed in multiple sectors, including emergency response, land management, marine sciences, and urban planning.

Problem

• Bridging gap in multi-modal remote sensing foundation models
• Handling diverse RS modalities with complementary insights
• Enabling efficient deployment via dynamic expert pruning

Innovation

• Hierarchical Mixture-of-Experts architecture for multi-modal learning
• Physics-informed self-supervised learning with radiometric characteristics
• Dynamic expert pruning for adaptive model compression