From Swath to Full-Disc: Advancing Precipitation Retrieval with Multimodal Knowledge Expansion

2025-06-08

AI Summary

Infrared remote sensing suffers from low precipitation retrieval accuracy and incomplete spatial coverage (limited to scan swaths), whereas passive microwave and radar sensors offer higher accuracy but suffer from limited spatiotemporal coverage. To address this, we propose a two-stage multimodal knowledge distillation framework. In Stage I, the Swath-Distilling paradigm leverages CoMWE (Collaborative Masked Wavelet Enhancement) to transfer high-fidelity precipitation information from passive microwave and radar sensors into an infrared-based model. In Stage II, the Full-Disc Adaptation paradigm, augmented by Self-MaskTune, enables cross-modal and cross-scale knowledge transfer and generalization across the entire geostationary disk. Built upon the PRE-Net backbone and multi-source remote sensing fusion data, our method achieves significant improvements over PERSIANN-CCS, PDIR, and IMERG on the newly established PRE benchmark, substantially enhancing full-disk precipitation estimation accuracy. The code is publicly available.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Accurate near-real-time precipitation retrieval has been enhanced by satellite-based technologies. However, infrared-based algorithms have low accuracy due to weak relations with surface precipitation, whereas passive microwave and radar-based methods are more accurate but limited in range. This challenge motivates the Precipitation Retrieval Expansion (PRE) task, which aims to enable accurate, infrared-based full-disc precipitation retrievals beyond the scanning swath. We introduce Multimodal Knowledge Expansion, a two-stage pipeline with the proposed PRE-Net model. In the Swath-Distilling stage, PRE-Net transfers knowledge from a multimodal data integration model to an infrared-based model within the scanning swath via Coordinated Masking and Wavelet Enhancement (CoMWE). In the Full-Disc Adaptation stage, Self-MaskTune refines predictions across the full disc by balancing multimodal and full-disc infrared knowledge. Experiments on the introduced PRE benchmark demonstrate that PRE-Net significantly advanced precipitation retrieval performance, outperforming leading products like PERSIANN-CCS, PDIR, and IMERG. The code will be available at https://github.com/Zjut-MultimediaPlus/PRE-Net.

Problem

• Enhance infrared-based full-disc precipitation retrieval accuracy
• Overcome limited range of microwave and radar methods
• Transfer multimodal knowledge to improve swath-to-disc predictions

Innovation

• Multimodal Knowledge Expansion pipeline
• Coordinated Masking and Wavelet Enhancement
• Self-MaskTune for full-disc refinement