Sensitivity-Informed Augmentation for Robust Segmentation

2024-06-03

arXiv.org

AI Summary

Image segmentation models suffer significant robustness degradation under real-world corruptions (e.g., adverse weather, sensor heterogeneity), while existing gradient-based natural corruption sensitivity analyses incur high computational overhead and exhibit poor generalization. Method: We propose a gradient-free, adaptive sensitivity-driven data augmentation framework, centered on a novel Adaptive Sensitivity Analysis (ASA) scheme based on Kernel Inception Distance (KID). ASA uniquely couples sensitivity curve modeling with dynamic online robustness re-evaluation, enabling end-to-end low-fine-tuning augmentation. The method integrates base perturbation analysis, adaptive hyperparameter sampling, and dynamic adversarial training. Results: Evaluated across multiple segmentation benchmarks, our approach consistently outperforms state-of-the-art augmentation methods, simultaneously improving accuracy on clean data and generalization under complex corruptions—without requiring architectural modifications or additional inference cost.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Segmentation is an integral module in many visual computing applications such as virtual try-on, medical imaging, autonomous driving, and agricultural automation. These applications often involve either widespread consumer use or highly variable environments, both of which can degrade the quality of visual sensor data, whether from a common mobile phone or an expensive satellite imaging camera. In addition to external noises like user difference or weather conditions, internal noises such as variations in camera quality or lens distortion can affect the performance of segmentation models during both development and deployment. In this work, we present an efficient, adaptable, and gradient-free method to enhance the robustness of learning-based segmentation models across training. First, we introduce a novel adaptive sensitivity analysis (ASA) using Kernel Inception Distance (KID) on basis perturbations to benchmark perturbation sensitivity of pre-trained segmentation models. Then, we model the sensitivity curve using the adaptive SA and sample perturbation hyperparameter values accordingly. Finally, we conduct adversarial training with the selected perturbation values and dynamically re-evaluate robustness during online training. Our method, implemented end-to-end with minimal fine-tuning required, consistently outperforms state-of-the-art data augmentation techniques for segmentation. It shows significant improvement in both clean data evaluation and real-world adverse scenario evaluation across various segmentation datasets used in visual computing and computer graphics applications.

Problem

• Enhancing robustness in image segmentation against natural corruptions
• Reducing computational cost of sensitivity analysis for corruptions
• Improving model-free augmentation policies for real-time applications

Innovation

• Adaptive sensitivity-guided augmentation method
• 10x faster sensitivity analysis
• 200x less storage required