OoDIS: Anomaly Instance Segmentation and Detection Benchmark

2024-06-17

AI Summary

Autonomous driving and robotics require reliable detection of unknown-category objects (e.g., wildlife, unconventional obstacles), yet existing out-of-distribution (OOD) perception research focuses predominantly on semantic segmentation, lacking standardized benchmarks for instance-level localization—hindering progress in anomaly instance segmentation and detection. Method: We introduce the first dedicated benchmark for anomaly instance segmentation and detection targeting unknown objects, extending OOD semantic segmentation benchmarks to the instance level for unified evaluation of both tasks. Leveraging Cityscapes, BDD100K, and other datasets, we provide expert-annotated anomaly instances and enhanced instance masks, along with a standardized evaluation protocol featuring fine-grained metrics (e.g., IoU, mAP) and an open competition platform. Contribution/Results: Experiments reveal that state-of-the-art methods achieve average mAP below 15%, underscoring the task’s difficulty. The benchmark is publicly released to foster community advancement in open-world perception.

Technology Category

• Computer Vision: Object Detection & Categorization
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Safe navigation of self-driving cars and robots requires a precise understanding of their environment. Training data for perception systems cannot cover the wide variety of objects that may appear during deployment. Thus, reliable identification of unknown objects, such as wild animals and untypical obstacles, is critical due to their potential to cause serious accidents. Significant progress in semantic segmentation of anomalies has been facilitated by the availability of out-of-distribution (OOD) benchmarks. However, a comprehensive understanding of scene dynamics requires the segmentation of individual objects, and thus the segmentation of instances is essential. Development in this area has been lagging, largely due to the lack of dedicated benchmarks. The situation is similar in object detection. While there is interest in detecting and potentially tracking every anomalous object, the availability of dedicated benchmarks is clearly limited. To address this gap, this work extends some commonly used anomaly segmentation benchmarks to include the instance segmentation and object detection tasks. Our evaluation of anomaly instance segmentation and object detection methods shows that both of these challenges remain unsolved problems. We provide a competition and benchmark website under https://vision.rwth-aachen.de/oodis

Problem

• Identifying unknown objects for safe autonomous navigation
• Lack of benchmarks for anomaly instance segmentation
• Limited benchmarks for anomaly object detection

Innovation

• Extends anomaly segmentation benchmarks for instance tasks
• Evaluates anomaly instance segmentation and detection methods
• Provides competition and benchmark website for OOD challenges