Graph-based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation and A Large-scale Benchmark

2026-04-29

AI Summary

This work addresses cross-modal spatial misalignment in UAV RGBT imagery—caused by sensor parallax and platform vibration—and semantic confusion among fine-grained land-cover categories under a top-down view. To tackle these challenges, the authors propose the Graph Semantic Calibration Network (GSCNet), which employs a feature disentanglement and alignment module for robust spatial correction. Furthermore, GSCNet innovatively embeds a structured category graph into a graph attention mechanism, incorporating hierarchical relationships and co-occurrence priors to calibrate predictions for visually similar and rare classes. The study introduces URTF, the first large-scale, unaligned fine-grained UAV RGBT segmentation benchmark, and demonstrates significant performance gains over existing methods across 61 fine-grained categories, particularly enhancing segmentation accuracy for easily confused and rare classes.

Abstract

Fine-grained RGBT image semantic segmentation is crucial for all-weather unmanned aerial vehicle (UAV) scene understanding. However, UAV RGBT semantic segmentation faces two coupled challenges: cross-modal spatial misalignment caused by sensor parallax and platform vibration, and severe semantic confusion among fine-grained ground objects under top-down aerial views. To address these issues, we propose a Graph-based Semantic Calibration Network (GSCNet) for unaligned UAV RGBT image semantic segmentation. Specifically, we design a Feature Decoupling and Alignment Module (FDAM) that decouples each modality into shared structural and private perceptual components and performs deformable alignment in the shared subspace, enabling robust spatial correction with reduced modality appearance interference. Moreover, we propose a Semantic Graph Calibration Module (SGCM) that explicitly encodes the hierarchical taxonomy and co-occurrence regularities among ground-object categories in UAV scenes into a structured category graph, and incorporates these priors into graph-attention reasoning to calibrate predictions of visually similar and rare categories.In addition, we construct the Unaligned RGB-Thermal Fine-grained (URTF) benchmark, to the best of our knowledge, the largest and most fine-grained benchmark for unaligned UAV RGBT image semantic segmentation, containing over 25,000 image pairs across 61 categories with realistic cross-modal misalignment. Extensive experiments on URTF demonstrate that GSCNet significantly outperforms state-of-the-art methods, with notable gains on fine-grained categories. The dataset is available at https://github.com/mmic-lcl/Datasets-and-benchmark-code.

Problem

• UAV RGBT semantic segmentation
• cross-modal misalignment
• semantic confusion
• fine-grained segmentation
• spatial alignment

Innovation

• Graph-based Semantic Calibration
• Feature Decoupling and Alignment
• Semantic Graph Reasoning
• Unaligned RGBT Segmentation
• UAV Benchmark