A Novel Large-scale Crop Dataset and Dual-stream Transformer Method for Fine-grained Hierarchical Crop Classification from Integrated Hyperspectral EnMAP Data and Multispectral Sentinel-2 Time Series

2025-06-06

AI Summary

Fine-grained crop classification faces challenges in jointly modeling multi-temporal phenological dynamics and nanoscale spectral variations. Method: We propose a dual-stream Transformer framework coupling a spectral-spatial Transformer (for EnMAP hyperspectral data) with a temporal Swin Transformer (for Sentinel-2 time-series imagery), integrated via a hierarchical fusion classification head supporting joint prediction across a four-level crop taxonomy. Contribution/Results: We introduce H2Crop—the first million-scale, fine-grained, multimodally aligned hyperspectral crop benchmark dataset. Experiments show our method improves mean F1-score by 4.2% (up to +6.3%) over a Sentinel-2-only baseline on the four-level classification task. Hyperspectral gains remain consistently significant across varying temporal windows and crop change scenarios, establishing a scalable, multi-source synergistic modeling paradigm for precision agriculture and food security monitoring.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Fine-grained crop classification is crucial for precision agriculture and food security monitoring. It requires simultaneous capture of both phenological dynamics (obtained from multi-temporal satellite data like Sentinel-2) and subtle spectral variations (demanding nanometer-scale spectral resolution from hyperspectral imagery). Research combining these two modalities remains scarce currently due to challenges in hyperspectral data acquisition and crop types annotation costs. To address these issues, we construct a hierarchical hyperspectral crop dataset (H2Crop) by integrating 30m-resolution EnMAP hyperspectral data with Sentinel-2 time series. With over one million annotated field parcels organized in a four-tier crop taxonomy, H2Crop establishes a vital benchmark for fine-grained agricultural crop classification and hyperspectral image processing. We propose a dual-stream Transformer architecture that synergistically processes these modalities. It coordinates two specialized pathways: a spectral-spatial Transformer extracts fine-grained signatures from hyperspectral EnMAP data, while a temporal Swin Transformer extracts crop growth patterns from Sentinel-2 time series. The designed hierarchy classification heads with hierarchical fusion then simultaneously delivers multi-level classification across all taxonomic tiers. Experiments demonstrate that adding hyperspectral EnMAP data to Sentinel-2 time series yields a 4.2% average F1-scores improvement (peaking at 6.3%). Extensive comparisons also confirming our method's higher accuracy over existing deep learning approaches for crop type classification and the consistent benefits of hyperspectral data across varying temporal windows and crop change scenarios. Codes and dataset will be available at https://github.com/flyakon/H2Crop and www.glass.hku.hk Keywords: Crop type classification, precision agriculture, remote sensing, deep learning, hyperspectral data, Sentinel-2 time series, fine-grained crops

Problem

• Develops a dual-stream Transformer for fine-grained crop classification
• Integrates hyperspectral and multispectral data to improve classification accuracy
• Addresses challenges in hyperspectral data acquisition and crop annotation costs

Innovation

• Dual-stream Transformer for multi-modal data fusion
• Hierarchical hyperspectral crop dataset (H2Crop) integration
• Hierarchical classification heads with multi-level fusion