A Joint Learning Framework with Feature Reconstruction and Prediction for Incomplete Satellite Image Time Series in Agricultural Semantic Segmentation

2025-05-25

AI Summary

Cloud cover causes missing and corrupted observations in Satellite Image Time Series (SITS), disrupting temporal dependencies and inducing feature shift. To address this, we propose a joint learning framework that synergistically optimizes feature reconstruction and semantic prediction. Our approach introduces a novel reconstruction–prediction dual-task coupling mechanism, constrained by temporally consistent features extracted from a full-sequence teacher model to prevent redundant reconstruction and noise propagation. We simulate arbitrary missing patterns via temporal masking and integrate knowledge distillation with multi-task optimization to enhance generalization. Evaluated on datasets from Hunan (China), Western France, and Catalonia (Spain), our method improves average F1 scores for cropland extraction and crop classification by 6.93% and 7.09%, respectively. It demonstrates robustness across heterogeneous sensors (Sentinel-2 and PlanetScope) and varying cloud-cover rates.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Computer Vision: Remote Sensing / Geospatial AI
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Satellite Image Time Series (SITS) is crucial for agricultural semantic segmentation. However, Cloud contamination introduces time gaps in SITS, disrupting temporal dependencies and causing feature shifts, leading to degraded performance of models trained on complete SITS. Existing methods typically address this by reconstructing the entire SITS before prediction or using data augmentation to simulate missing data. Yet, full reconstruction may introduce noise and redundancy, while the data-augmented model can only handle limited missing patterns, leading to poor generalization. We propose a joint learning framework with feature reconstruction and prediction to address incomplete SITS more effectively. During training, we simulate data-missing scenarios using temporal masks. The two tasks are guided by both ground-truth labels and the teacher model trained on complete SITS. The prediction task constrains the model from selectively reconstructing critical features from masked inputs that align with the teacher's temporal feature representations. It reduces unnecessary reconstruction and limits noise propagation. By integrating reconstructed features into the prediction task, the model avoids learning shortcuts and maintains its ability to handle varied missing patterns and complete SITS. Experiments on SITS from Hunan Province, Western France, and Catalonia show that our method improves mean F1-scores by 6.93% in cropland extraction and 7.09% in crop classification over baselines. It also generalizes well across satellite sensors, including Sentinel-2 and PlanetScope, under varying temporal missing rates and model backbones.

Problem

• Addresses cloud-contaminated gaps in satellite image time series
• Reduces noise by selective feature reconstruction and prediction
• Improves agricultural segmentation accuracy across varied missing patterns

Innovation

• Joint learning with feature reconstruction and prediction
• Temporal masks simulate data-missing scenarios
• Teacher model guides selective feature reconstruction