A Causally Informed Pretraining Approach for Multimodal Foundation Models: Applications in Remote Sensing

2024-07-29

AI Summary

Existing remote sensing pretraining methods—such as masked image modeling and next-token prediction—neglect causal relationships among geospatial environmental variables, limiting interpretability and generalization in downstream tasks. To address this, we propose Causal-Driven Variable-Step Forecasting (CI-VSF), the first pretraining paradigm that explicitly embeds causal structure into multimodal foundation models: it conditions on driving variables (e.g., meteorological data) to generatively model response variables (e.g., satellite imagery), enabling conditional generative pretraining. Our approach integrates causal modeling, conditional variational autoencoding, and multimodal temporal alignment to jointly learn representations of remote sensing images and meteorological time series. Evaluated on crop mapping, missing image reconstruction, and soil moisture estimation, CI-VSF achieves average accuracy gains of 3.2–7.8% over baselines including MAE and SimCLR. Results demonstrate substantial improvements in interpretability, cross-task transferability, and out-of-distribution generalization afforded by causal pretraining.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Causal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Self-supervised learning has emerged as a powerful paradigm for pretraining foundation models using large-scale data. Existing pretraining approaches predominantly rely on masked reconstruction or next-token prediction strategies, demonstrating strong performance across various downstream tasks, including geoscience applications. However, these approaches do not fully capture the causal interplay between different geospatial and environmental variables. To address this limitation, we propose Causally Informed Variable-Step Forecasting (CI-VSF), a novel pretraining task that models forecasting as a conditional generation task, where driver variables (e.g., weather) inform the prediction of response variables (e.g., satellite imagery). We demonstrate that pretraining in such a fashion leads to enhanced performance when finetuned on both prediction (e.g., crop mapping, missing image prediction, soil moisture estimation) and forecasting (e.g., future image forecasting, soil moisture forecasting) downstream tasks when compared to other pretraining approaches. While we use remote sensing as our main application to demonstrate the efficacy of our proposed pretraining strategy over existing paradigms, it is applicable to any domain that involves known causal relationships amongst a set of variables.

Problem

• Enhancing causal interplay capture in pretraining
• Improving prediction and forecasting in remote sensing
• Applying causal relationships to multimodal foundation models

Innovation

• Causally Informed Pretraining Approach
• Conditional Generation Task
• Enhanced Performance in Forecasting Tasks