Hyperspectral Image Land Cover Captioning Dataset for Vision Language Models

2025-05-18

AI Summary

Existing hyperspectral imaging (HSI) datasets are limited to coarse-grained classification and lack pixel-level semantic understanding. To address this, we introduce HyperCap—the first large-scale hyperspectral scene captioning dataset—enabling fine-grained, pixel-level textual descriptions of spectral imagery. We propose a hybrid annotation paradigm combining automated pre-screening with expert verification, covering four major HSI benchmarks and bridging the critical gap in large-scale vision-language alignment for remote sensing. Leveraging multi-source encoders—ViT for spatial features and SpectralFormer for spectral features—we systematically evaluate early/late fusion and cross-attention mechanisms. Experiments demonstrate that text-guided supervision significantly improves classification accuracy (average +5.2%), establishing a new benchmark and methodological foundation for hyperspectral vision-language modeling.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web

Abstract

We introduce HyperCap, the first large-scale hyperspectral captioning dataset designed to enhance model performance and effectiveness in remote sensing applications. Unlike traditional hyperspectral imaging (HSI) datasets that focus solely on classification tasks, HyperCap integrates spectral data with pixel-wise textual annotations, enabling deeper semantic understanding of hyperspectral imagery. This dataset enhances model performance in tasks like classification and feature extraction, providing a valuable resource for advanced remote sensing applications. HyperCap is constructed from four benchmark datasets and annotated through a hybrid approach combining automated and manual methods to ensure accuracy and consistency. Empirical evaluations using state-of-the-art encoders and diverse fusion techniques demonstrate significant improvements in classification performance. These results underscore the potential of vision-language learning in HSI and position HyperCap as a foundational dataset for future research in the field.

Problem

• Enhancing model performance in hyperspectral image captioning
• Integrating spectral data with textual annotations for semantic understanding
• Improving classification and feature extraction in remote sensing

Innovation

• Integrates spectral data with textual annotations
• Uses hybrid automated and manual annotation
• Applies vision-language learning to hyperspectral imagery