Unlocking Robust Semantic Segmentation Performance via Label-only Elastic Deformations against Implicit Label Noise

2025-08-14

AI Summary

Implicit label noise—such as ambiguous boundaries and inter-annotator discrepancies—is prevalent in real-world semantic segmentation datasets. Conventional synchronous image-label augmentation exacerbates such noise, degrading model generalization. To address this, we propose a decoupled label augmentation framework that, for the first time, applies elastic deformation exclusively within the segmentation label space—breaking the conventional paradigm of joint image-label geometric transformation. This design enhances model robustness to minor annotation inconsistencies without requiring auxiliary modules or preprocessing, enabling seamless integration into standard training pipelines. Extensive experiments demonstrate consistent improvements: +2.29, +2.38, +1.75, and +3.39 average mIoU gains on Vaihingen, LoveDA, Cityscapes, and PASCAL VOC, respectively. Moreover, our method exhibits complementary benefits when combined with existing techniques such as CutMix and label smoothing.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

While previous studies on image segmentation focus on handling severe (or explicit) label noise, real-world datasets also exhibit subtle (or implicit) label imperfections. These arise from inherent challenges, such as ambiguous object boundaries and annotator variability. Although not explicitly present, such mild and latent noise can still impair model performance. Typical data augmentation methods, which apply identical transformations to the image and its label, risk amplifying these subtle imperfections and limiting the model's generalization capacity. In this paper, we introduce NSegment+, a novel augmentation framework that decouples image and label transformations to address such realistic noise for semantic segmentation. By introducing controlled elastic deformations only to segmentation labels while preserving the original images, our method encourages models to focus on learning robust representations of object structures despite minor label inconsistencies. Extensive experiments demonstrate that NSegment+ consistently improves performance, achieving mIoU gains of up to +2.29, +2.38, +1.75, and +3.39 in average on Vaihingen, LoveDA, Cityscapes, and PASCAL VOC, respectively-even without bells and whistles, highlighting the importance of addressing implicit label noise. These gains can be further amplified when combined with other training tricks, including CutMix and Label Smoothing.

Problem

• Addresses implicit label noise in semantic segmentation
• Decouples image and label transformations for robustness
• Improves model performance despite minor label inconsistencies

Innovation

• Decouples image and label transformations
• Applies elastic deformations only to labels
• Improves robustness against label noise