Subgraph-Oriented Testing for Deep Learning Libraries

2024-12-09

arXiv.org

AI Summary

Existing deep learning library testing methods suffer from three key limitations: (1) difficulty in localizing faults at the model level; (2) API-level testing that frequently generates invalid inputs, overemphasizes crash detection while neglecting accuracy defects, and ignores realistic API interactions; and (3) insufficient modeling of cross-API dependencies. This paper proposes a subgraph-guided realistic interaction testing paradigm. It treats high-frequency subgraphs extracted from PyTorch computational graphs as test units, integrating frequent subgraph mining for behavioral modeling, runtime-feature-driven input generation, cross-platform differential execution for oracle-free validation, and graph-parsing-based replay for precise fault localization. The approach ensures high-fidelity, realistic inputs while enabling accurate identification of accuracy-related faults. Evaluated on 728 high-frequency subgraphs across 49 mainstream models, it achieves 100% valid input generation and uncovers 18 previously unknown accuracy bugs—marking the first systematic discovery of interaction-induced accuracy defects arising from API co-invocation.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

Deep Learning (DL) libraries, such as PyTorch, are widely used for building and deploying DL models on various hardware platforms. Meanwhile, they are found to contain bugs that lead to incorrect calculation results and cause issues like non-convergence training and inaccurate prediction of DL models. Thus, many efforts have been made to test DL libraries and reveal bugs. However, existing DL library testing methods manifest limitations: model-level testing methods cause complexity in fault localization. Meanwhile, API-level testing methods often generate invalid inputs or primarily focus on extreme inputs that lead to crash failures; they also ignore testing realistic API interactions. These limitations may lead to missing detection of bugs, even in the frequently used APIs. To address these limitations, we propose SORT (Subgraph-Oriented Realistic Testing) to differential test DL libraries on different hardware platforms. SORT takes popular API interaction patterns, represented as frequent subgraphs of model computation graphs, as test subjects. In this way, it introduces realistic API interaction sequences while maintaining efficiency in locating faulty APIs for observed errors. Besides, SORT prepares test inputs by referring to extensive features of runtime inputs for each API in executing real-life benchmark data. The generated inputs are expected to better simulate such valid real inputs and reveal bugs more likely to happen in real-life usage. Evaluation on 728 frequent subgraphs of 49 popular PyTorch models demonstrates that SORT achieves a 100% valid input generation rate, detects more precision bugs than existing methods, and reveals interaction-related bugs missed by single-API testing. 18 precision bugs in PyTorch are identified.

Problem

• Testing DL libraries for bugs in API interactions
• Generating valid inputs to simulate real-life usage
• Improving fault localization in DL library testing

Innovation

• Subgraph-Oriented Testing for realistic API interactions
• Differential testing across different hardware platforms
• Valid input generation from real-life benchmark data