SpecDM: Hyperspectral Dataset Synthesis with Pixel-level Semantic Annotations

2025-02-24

AI Summary

In hyperspectral remote sensing, dense prediction tasks—such as semantic segmentation and change detection—are severely hindered by the scarcity and high acquisition cost of pixel-level ground-truth annotations. To address this, we propose the first synthesis method for pixel-level semantic labeling of hyperspectral images (HSIs). Our approach introduces a novel dual-stream variational autoencoder (VAE) coupled with a conditional diffusion model, jointly modeling the latent-space joint distribution of HSIs and their corresponding semantic masks to enable co-generation of high-dimensional spectral imagery and precise pixel-wise labels. This is the first framework capable of end-to-end, semantic-mask-conditioned HSI generation. Evaluated on multiple benchmarks, our synthesized data significantly improves downstream model performance—achieving accuracy comparable to models trained on real annotated data—thereby demonstrating both effectiveness and strong generalization capability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

In hyperspectral remote sensing field, some downstream dense prediction tasks, such as semantic segmentation (SS) and change detection (CD), rely on supervised learning to improve model performance and require a large amount of manually annotated data for training. However, due to the needs of specific equipment and special application scenarios, the acquisition and annotation of hyperspectral images (HSIs) are often costly and time-consuming. To this end, our work explores the potential of generative diffusion model in synthesizing HSIs with pixel-level annotations. The main idea is to utilize a two-stream VAE to learn the latent representations of images and corresponding masks respectively, learn their joint distribution during the diffusion model training, and finally obtain the image and mask through their respective decoders. To the best of our knowledge, it is the first work to generate high-dimensional HSIs with annotations. Our proposed approach can be applied in various kinds of dataset generation. We select two of the most widely used dense prediction tasks: semantic segmentation and change detection, and generate datasets suitable for these tasks. Experiments demonstrate that our synthetic datasets have a positive impact on the improvement of these downstream tasks.

Problem

• Synthesize hyperspectral images with annotations
• Reduce cost and time in data acquisition
• Improve semantic segmentation and change detection tasks

Innovation

• Generative diffusion model for HSI synthesis
• Two-stream VAE for latent representations
• Pixel-level annotated dataset generation