Distribution-free false-alarm calibration and chance-corrected spatial evaluation for industrial anomaly detection

2026-08-15

AI Summary

This study addresses the limitations of conventional metrics in industrial anomaly detection, specifically their inability to specify false positive rates and their tendency to yield inflated overlap scores. To overcome these issues, we propose a distribution-free false positive calibration framework alongside an opportunity-corrected space evaluation method. By employing tolerance thresholds, paired-subtraction cross-validation, and permutation tests, this approach quantifies genuine spatial evidence improvement after eliminating random overlaps, thereby revealing the intrinsic bias of traditional metrics. Experimental results demonstrate that the proposed method significantly enhances the assessment of detector performance. Furthermore, it provides sample size planning guidelines determining the minimum number of normal samples required to support specific false positive rates, establishing a more rigorous evaluation paradigm and sampling basis for industrial visual inspection.

Abstract

Studies of industrial visual inspection commonly report the area under the receiver operating characteristic curve (AUROC) and the overlap between anomaly maps and defect masks. Neither measure specifies the false-alarm rate at a selected threshold, while recurrent defect locations and mask geometry can inflate overlap. We combine a distribution-free upper tolerance threshold with a paired-minus-crossed spatial test. This test compares each detector's score-contributing locations with the matched defect mask and with masks from other images; the difference in rates defines spatial-evidence lift relative to the empirical chance-overlap rate. We evaluate three detectors on 120 point-defect images from three ISP-AD modalities and three fixed data splits. Of 378 alarms, 230 overlap the matched mask. Paired and crossed rates are nevertheless similar in eight of nine detector--modality cells; only DINOv2--ASM has a positive 95\% bootstrap lower bound (lift 0.259, 95\% interval 0.159--0.347). On the independent Magnetic Tile Defect dataset, the same analysis gives lifts of 0.203 (0.169--0.236) for Wide ResNet-50 (WRN50) patch memory and 0.231 (0.202--0.262) for Vision Transformer B/16 (ViT-B/16) patch memory, with one-sided permutation $p=10^{-5}$ for both. When crossed masks are restricted to the same defect class, the lifts remain 0.185 and 0.210. Exact sample planning shows that, with 150 calibration normals, a 95\%-confidence distribution-free claim is supported only for target false-positive rates of 1.98\% or higher; a 1\% target requires at least 299 normals. The results support reporting operating-point performance and chance-corrected spatial evidence alongside AUROC and raw mask overlap.

Problem

• Industrial Anomaly Detection
• False-alarm Calibration
• Spatial Evaluation
• Chance-corrected Metrics
• Mask Overlap

Innovation

• Distribution-free calibration
• Chance-corrected spatial evaluation
• Spatial-evidence lift
• Paired-minus-crossed test
• Industrial anomaly detection