Automated Model Evaluation for Object Detection via Prediction Consistency and Reliablity

2025-08-16

AI Summary

Existing object detection evaluation heavily relies on costly manual annotations. This paper proposes the first label-free automated evaluation framework, which estimates detection performance in an unsupervised manner by modeling spatial consistency and confidence reliability of predictions before and after non-maximum suppression (NMS). Our method innovatively leverages geometric overlap and confidence distribution consistency among internal candidate bounding boxes as intrinsic evaluation signals. To enhance generalizability and robustness, we construct a metadata set covering diverse image degradation types. Experiments across multiple benchmarks demonstrate that our approach significantly outperforms existing automatic evaluation methods—reducing performance prediction error by up to 32%—and accurately captures model behavior shifts under varying degradation levels. The source code is publicly available.

Technology Category

• Computer Vision: Object Detection & Categorization
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Recent advances in computer vision have made training object detectors more efficient and effective; however, assessing their performance in real-world applications still relies on costly manual annotation. To address this limitation, we develop an automated model evaluation (AutoEval) framework for object detection. We propose Prediction Consistency and Reliability (PCR), which leverages the multiple candidate bounding boxes that conventional detectors generate before non-maximum suppression (NMS). PCR estimates detection performance without ground-truth labels by jointly measuring 1) the spatial consistency between boxes before and after NMS, and 2) the reliability of the retained boxes via the confidence scores of overlapping boxes. For a more realistic and scalable evaluation, we construct a meta-dataset by applying image corruptions of varying severity. Experimental results demonstrate that PCR yields more accurate performance estimates than existing AutoEval methods, and the proposed meta-dataset covers a wider range of detection performance. The code is available at https://github.com/YonseiML/autoeval-det.

Problem

• Automates object detector evaluation without manual annotation
• Measures prediction consistency and reliability for performance estimation
• Uses meta-dataset with image corruptions for realistic evaluation

Innovation

• Automated model evaluation via prediction consistency
• Reliability assessment using confidence scores
• Meta-dataset with varying image corruptions