COARSE: Collaborative Pseudo-Labeling with Coarse Real Labels for Off-Road Semantic Segmentation

2025-03-05

AI Summary

In off-road autonomous driving, semantic segmentation suffers from scarce annotations: real-world data provides only sparse, coarse-grained labels, while synthetic data offers dense labels but exhibits significant domain shift. To address this, we propose a semi-supervised domain adaptation framework. Our method employs a dual-decoder architecture—operating at both pixel- and patch-level—to collaboratively generate pseudo-labels by fusing intra-domain coarse annotations with inter-domain fine-grained labels. It leverages a pre-trained Vision Transformer (ViT) backbone, cross-domain consistency regularization, and an iterative co-training pseudo-labeling strategy to mitigate the simulation-to-reality distribution gap. Evaluated on RUGD and Rellis-3D benchmarks, our approach achieves absolute mIoU improvements of 9.7% and 8.4%, respectively. Furthermore, extensive real-world deployment across diverse biomes demonstrates strong generalization and robustness.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health

Abstract

Autonomous off-road navigation faces challenges due to diverse, unstructured environments, requiring robust perception with both geometric and semantic understanding. However, scarce densely labeled semantic data limits generalization across domains. Simulated data helps, but introduces domain adaptation issues. We propose COARSE, a semi-supervised domain adaptation framework for off-road semantic segmentation, leveraging sparse, coarse in-domain labels and densely labeled out-of-domain data. Using pretrained vision transformers, we bridge domain gaps with complementary pixel-level and patch-level decoders, enhanced by a collaborative pseudo-labeling strategy on unlabeled data. Evaluations on RUGD and Rellis-3D datasets show significant improvements of 9.7% and 8.4% respectively, versus only using coarse data. Tests on real-world off-road vehicle data in a multi-biome setting further demonstrate COARSE's applicability.

Problem

• Addresses domain adaptation in off-road semantic segmentation.
• Utilizes sparse coarse labels and dense out-of-domain data.
• Improves segmentation accuracy in diverse, unstructured environments.

Innovation

• Semi-supervised domain adaptation framework
• Collaborative pseudo-labeling strategy
• Pixel-level and patch-level decoders