Fi-ImageNet-1k: An OOD Benchmark From the Inside of the ImageNet-1k Validation Set

2026-09-01

AI Summary

本文通过利用ImageNet-1k验证集中的标注错误，构建了一个新的OOD数据集Fi-ImageNet-1k，用于评估模型在识别不属于预定义类别的图像时的表现。

Abstract

Out-of-distribution (OOD) detection predicts whether a test image belongs to none of the predefined classes. To evaluate this task, benchmarks need images from outside the in-distribution (ID) data; typically, these are defined or collected in an ad hoc fashion. Since no ground truth is perfect, ID-labeled datasets themselves contain a natural source of OOD images. We exploit such annotation errors and present Fi-ImageNet-1k, an OOD dataset built from ImageNet-1k validation images that the recent ReImageNet reannotation effort assigned to no ImageNet-1k class. Each image was examined by expert human annotators supported by evidence from MLLMs, VLMs, and reverse image search, comparing it against all visually similar ID classes. We keep only images that could be assigned a specific class outside the ImageNet-1k label space. The resulting Fi-ImageNet-1k, with 655 images from 522 classes, is substantially more challenging than any commonly used OOD dataset. No evaluated combination of classifier and OOD detector achieves a false positive rate below 51% at 95% true positive rate (FPR@95). Compared to the recent NINCO, our dataset is 3.8x more challenging in the FPR@95 metric for state-of-the-art supervised OOD detection methods.

Problem

• Out-of-distribution
• benchmark
• ImageNet-1k
• annotation errors
• evaluation

Innovation

• OOD Detection
• Annotation Errors
• Reannotation Effort
• Expert Human Annotators
• MLLMs and VLMs