Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors

2026-06-22

AI Summary

This work addresses the challenge of adapting remote sensing models to new sensor modalities following satellite sensor updates or replacements, a process typically hindered by the high cost of re-annotation and retraining. To this end, the authors propose DeluluNet, which for the first time systematically formalizes modality shifts in remote sensing into three canonical scenarios: replacement, addition, and subset selection. DeluluNet features an end-to-end trainable modular architecture that employs a modality hallucination mechanism to infer representations of missing modalities from available ones. By integrating a teacher–student framework with multimodal representation learning, the method leverages unlabeled data to enable cross-modal adaptation without requiring additional annotations. Experimental results demonstrate that DeluluNet maintains strong predictive performance under dynamically varying input modalities, substantially reducing reliance on full-scale retraining.

Abstract

Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled multimodal data via modality hallucination--predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.

Problem

• remote sensing
• modality adaptation
• satellite sensors
• machine learning
• changing modalities

Innovation

• modality hallucination
• DeluluNet
• modality transfer
• remote sensing
• multimodal learning