Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencoder

2026-04-18

AI Summary

This work addresses the challenges of representation degradation and negative transfer arising from the coexistence of modality heterogeneity and high resolution in joint pretraining of optical and SAR imagery. To this end, the authors propose CoDe-MAE, a novel framework that introduces the “minimal alignment, optimal collaboration” paradigm. It integrates three core techniques—Optical-Anchored Knowledge Distillation (OKD), Conditional Contrastive Learning (CCL), and Cross-modal Degradation Reconstruction (CDR)—to preserve inherent physical differences while uncovering shared semantic structures. Remarkably, with only 1M pretraining samples, CoDe-MAE surpasses larger foundation models and achieves new state-of-the-art performance across diverse downstream tasks, including both single-modality and multimodal settings, thereby significantly enhancing data efficiency and representation robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Learning robust representations across extremely heterogeneous modalities remains a fundamental challenge in multi-modal vision. As a critical and profound instantiation of this challenge, high-resolution (HR) joint optical and synthetic aperture radar (SAR) pretraining seeks modality synergy to mutually enhance single-source representations; its potential is severely hindered by the Heterogeneity-Resolution Paradox: finer spatial scales drastically amplify the physical divergence between complex radar geometries and non-homologous optical textures. Consequently, migrating medium-resolution-oriented rigid alignment paradigms to HR scenarios triggers either severe feature suppression to force equivalence, or feature contamination driven by extreme epistemic uncertainty. Both extremes inevitably culminate in profound representation degradation and negative transfer. To overcome this bottleneck, we propose CoDe-MAE, pioneering a \textit{better synergy with less alignment} philosophy. First, Optical-anchored Knowledge Distillation (OKD) implicitly regularizes SAR's speckle noise by mapping it into a pure semantic manifold. Building on this, Conditioned Contrastive Learning (CCL) utilizes a gradient buffering mechanism to align shared consensus while safely preserving divergent physical signatures. Concurrently, Cross-Modal Degraded Reconstruction (CDR) deliberately strips non-homologous spectral pseudo-features, truncating the inherently ill-posed mapping to capture true structural invariants. Extensive analyses validate our theoretical claims. Pretrained on 1M samples, CoDe-MAE demonstrates remarkable data efficiency, successfully preventing representation degradation and establishing new state-of-the-art performance across diverse single- and bi-modal downstream tasks, substantially outperforming foundation models scaled on vastly larger datasets.

Problem

• heterogeneous multi-modal
• high-resolution joint pretraining
• optical and SAR images
• representation degradation
• Heterogeneity-Resolution Paradox

Innovation

• Masked Autoencoder
• Heterogeneous Multi-Modal Learning
• Synthetic Aperture Radar
• Conditioned Contrastive Learning
• Cross-Modal Degraded Reconstruction