Confidence Scores in Open-Vocabulary Detection Are a Biased Mixture of Scale and Semantics

2026-07-12

AI Summary

This study addresses the confounding influence of object scale and semantic query specificity on confidence scores in open-vocabulary object detection, which hinders accurate reflection of true detection quality. Through systematic experiments on COCO and LVIS with models including GroundingDINO, OWL-ViT, and YOLO-World, the work reveals—for the first time—that image-level pretraining inherently induces a dual bias structure coupling scale and semantics. To mitigate this, the authors propose a training-free temperature scaling calibration method. Experiments show that this approach improves Recall@10 for small objects by 19.6% (p<0.01), while oracle thresholding yields a substantially larger F1 gain for large objects (Δ=0.102) compared to small ones (Δ=0.001). However, calibration incurs measurable ranking accuracy degradation, demonstrating that such biases can only be partially and non-losslessly alleviated at inference time.

Abstract

Foundation models such as CLIP have enabled open-vocabulary object detectors that generalise to novel categories via vision-language similarity. However, the confidence scores these detectors produce are not reliable localization probability estimates: they conflate visual scale and semantic query specificity with the true detection signal. Through controlled experiments on COCO across three foundation-model-based detectors (GroundingDINO, OWL-ViT, YOLO-World), with the scale-bias finding further replicated on LVIS (1,203 categories) using GroundingDINO, we show that s=cos(v,t) is a biased mixture of two effects. Scale bias (alpha = +0.064, r = 0.579, p = 1.29 x 10^-58) systematically inflates scores for large objects. Semantic bias (beta = -0.705, p = 5.23 x 10^-41) suppresses scores for generic queries. Both biases are structurally inevitable from CLIP's image-level pretraining. Threshold adjustment cannot remove them: oracle per-scale thresholding yields Delta F1 = +0.001 for small objects versus +0.102 for large. A parameter-free temperature scaling correction improves small-object Recall@10 by 19.6% (p < 0.01) without retraining. This comes at a modest, measurable cost to pooled-ranking precision, so the bias is partially, not freely, reversible at inference time. These findings reveal a fundamental limitation of adapting image-level foundation models to region-level detection tasks.

Problem

• open-vocabulary detection
• confidence scores
• scale bias
• semantic bias
• foundation models

Innovation

• confidence bias
• open-vocabulary detection
• scale bias
• semantic bias
• temperature scaling