Revisiting"Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion": A Critical Review and Implications on DNN Coverage Testing

2026-01-13

AI Summary

This study addresses significant theoretical and empirical shortcomings in the recently proposed Neural Coverage (NLC) criterion for deep neural network (DNN) testing. We present the first systematic reproduction and critical evaluation of NLC, originally introduced at ICSE 2023, demonstrating through rigorous theoretical analysis and empirical validation that it violates fundamental properties expected of coverage criteria—namely monotonicity and invariance to test suite ordering—and fails to adequately exploit the structural information embedded in covariance matrices. Our findings reveal critical validity threats and limitations in NLC’s underlying assumptions, empirically confirming its failure to satisfy essential coverage attributes. Building on these insights, we propose principled directions for improvement by incorporating richer covariance structure into coverage design. This work offers foundational guidance toward developing more reliable and theoretically sound DNN testing coverage metrics.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Evaluation and Analysis
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

We present a critical review of Neural Coverage (NLC), a state-of-the-art DNN coverage criterion by Yuan et al. at ICSE 2023. While NLC proposes to satisfy eight design requirements and demonstrates strong empirical performance, we question some of their theoretical and empirical assumptions. We observe that NLC deviates from core principles of coverage criteria, such as monotonicity and test suite order independence, and could more fully account for key properties of the covariance matrix. Additionally, we note threats to the validity of the empirical study, related to the ground truth ordering of test suites. Through our empirical validation, we substantiate our claims and propose improvements for future DNN coverage metrics. Finally, we conclude by discussing the implications of these insights.

Problem

• Neuron Coverage
• DNN Testing
• Coverage Criteria
• Empirical Validation
• Test Suite Evaluation

Innovation

• Neuron Coverage
• DNN Testing
• Coverage Criterion
• Covariance Matrix
• Empirical Validation