DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications

2025-02-24

AI Summary

Existing self-supervised multimodal methods in Earth observation produce coarse-grained (patch-level) embeddings, hindering precise alignment with pixel-level LiDAR data and limiting cross-modal fusion and fine-grained applications. This paper introduces the first pixel-level cross-modal alignment framework for remote sensing imagery and full-waveform LiDAR, leveraging contrastive learning to align image-LiDAR representations at the pixel level, thereby constructing a unified embedding space that supports zero-shot transfer and few-shot fine-tuning. Key contributions include: (1) the first pixel-level image-LiDAR alignment paradigm; (2) end-to-end modeling of full-waveform LiDAR; and (3) a zero-shot classifier adaptation mechanism. Experiments demonstrate that our method achieves superior zero-shot performance over task-specific supervised models across seven environmental monitoring tasks; under few-shot fine-tuning, it matches or surpasses state-of-the-art methods on five of six evaluated tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Significant efforts have been directed towards adapting self-supervised multimodal learning for Earth observation applications. However, existing methods produce coarse patch-sized embeddings, limiting their effectiveness and integration with other modalities like LiDAR. To close this gap, we present DUNIA, an approach to learn pixel-sized embeddings through cross-modal alignment between images and full-waveform LiDAR data. As the model is trained in a contrastive manner, the embeddings can be directly leveraged in the context of a variety of environmental monitoring tasks in a zero-shot setting. In our experiments, we demonstrate the effectiveness of the embeddings for seven such tasks (canopy height mapping, fractional canopy cover, land cover mapping, tree species identification, plant area index, crop type classification, and per-pixel waveform-based vertical structure mapping). The results show that the embeddings, along with zero-shot classifiers, often outperform specialized supervised models, even in low data regimes. In the fine-tuning setting, we show strong low-shot capabilities with performances near or better than state-of-the-art on five out of six tasks.

Problem

• Learn pixel-sized embeddings
• Cross-modal alignment for Earth observation
• Enhance environmental monitoring tasks

Innovation

• Pixel-sized embeddings via cross-modal alignment
• Contrastive training for zero-shot tasks
• Strong low-shot capabilities in fine-tuning