AdaStop: Cost-Aware Early Stopping for DNN Test Selection

2026-07-05

AI Summary

This work addresses the challenge of efficiently uncovering model failures in deep neural network testing under limited annotation budgets. It formalizes the test stopping problem as a cost-aware dynamic decision process and introduces AdaStop, an adaptive stopping mechanism that continuously estimates the marginal fault discovery rate and terminates annotation early when this rate falls below a predefined cost-benefit threshold. Integrated with diverse test input selection strategies and evaluated within an empirical framework across multiple datasets and models, AdaStop achieves 65%–84% fault detection using only 9%–31% of the annotation budget, substantially improving testing efficiency while significantly reducing labeling costs.

Abstract

Existing methods for testing deep neural networks (DNNs) primarily prioritize test inputs likely to reveal model faults under a fixed labeling budget. In practice, choosing that budget is difficult: too little testing misses failures, while too much incurs unnecessary labeling costs. This work studies the stopping problem in DNN testing. We formulate testing as a cost--benefit decision process in which labeling an input incurs cost $c$ and discovering a fault yields value $v$. Based on this formulation, we introduce \textit{AdaStop}, a framework that estimates the marginal fault discovery rate during testing and stops labeling when the estimated rate falls below the threshold $τ= c/v$. Experiments across multiple datasets, architectures, and selection strategies show that $65$--$84\%$ of faults can be discovered using only $9$--$31\%$ of the labeling budget.

Problem

• early stopping
• DNN testing
• labeling cost
• fault discovery
• cost-aware

Innovation

• early stopping
• cost-aware testing
• fault discovery rate
• DNN testing
• adaptive stopping