A Dataset for Semantic Segmentation in the Presence of Unknowns

2025-03-28

AI Summary

Existing semantic segmentation benchmarks lack the capability to jointly evaluate model performance on both known classes and unknown anomalies—particularly critical for safety-critical applications like autonomous driving. Method: We introduce ISSU, the first benchmark enabling unified pixel-level evaluation of known classes and unknown anomalies in semantic segmentation. ISSU features real-world cluttered scenes with multi-sensor inputs, diverse illumination conditions, and domain shifts, and includes both static image and video test sets. Contribution/Results: ISSU pioneers a spatio-temporal dual-modality testing protocol, fine-grained anomaly attribution, and a cross-domain generalization quantification framework. With twice the scale of prior anomaly segmentation benchmarks, ISSU reveals substantial performance degradation of state-of-the-art methods under cross-domain settings and on small- or large-scale anomalies—demonstrating its necessity and value as a rigorous benchmark for robust open-world segmentation research.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Before deployment in the real-world deep neural networks require thorough evaluation of how they handle both knowns, inputs represented in the training data, and unknowns (anomalies). This is especially important for scene understanding tasks with safety critical applications, such as in autonomous driving. Existing datasets allow evaluation of only knowns or unknowns - but not both, which is required to establish"in the wild"suitability of deep neural network models. To bridge this gap, we propose a novel anomaly segmentation dataset, ISSU, that features a diverse set of anomaly inputs from cluttered real-world environments. The dataset is twice larger than existing anomaly segmentation datasets, and provides a training, validation and test set for controlled in-domain evaluation. The test set consists of a static and temporal part, with the latter comprised of videos. The dataset provides annotations for both closed-set (knowns) and anomalies, enabling closed-set and open-set evaluation. The dataset covers diverse conditions, such as domain and cross-sensor shift, illumination variation and allows ablation of anomaly detection methods with respect to these variations. Evaluation results of current state-of-the-art methods confirm the need for improvements especially in domain-generalization, small and large object segmentation.

Problem

• Evaluates deep neural networks on known and unknown inputs
• Addresses lack of datasets for combined known-unknown evaluation
• Improves anomaly segmentation in diverse real-world conditions

Innovation

• Novel anomaly segmentation dataset ISSU
• Diverse anomaly inputs from real-world
• Closed-set and open-set evaluation annotations