Deep Learning Library Testing: Definition, Methods and Challenges

2024-04-27

AI Summary

Deep learning (DL) infrastructure libraries—including frameworks, compilers, and hardware-accelerator libraries—lack systematic definitions and methodological foundations for testing. Method: This paper presents the first comprehensive study on DL library testing: (1) it proposes a novel three-category taxonomy of DL libraries and corresponding defect definitions, unifying the scope and boundaries of DL library defects and testing; (2) it introduces the first end-to-end testing methodology framework covering the full DL stack, with a structured survey of domain-specific testing techniques and tools; and (3) it systematically identifies six cross-cutting testing challenges and outlines concrete directions for future research. Contribution/Results: Based on rigorous literature review, taxonomic analysis, and comparative evaluation, this work establishes the first authoritative technical guideline for DL library testing—providing industry practitioners with actionable, deployable testing strategies and offering academia a benchmark paradigm for further investigation.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Description Logics
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In recent years, software systems powered by deep learning (DL) techniques have significantly facilitated people's lives in many aspects. As the backbone of these DL systems, various DL libraries undertake the underlying optimization and computation. However, like traditional software, DL libraries are not immune to bugs, which can pose serious threats to users' personal property and safety. Studying the characteristics of DL libraries, their associated bugs, and the corresponding testing methods is crucial for enhancing the security of DL systems and advancing the widespread application of DL technology. This paper provides an overview of the testing research related to various DL libraries, discusses the strengths and weaknesses of existing methods, and provides guidance and reference for the application of the DL library. This paper first introduces the workflow of DL underlying libraries and the characteristics of three kinds of DL libraries involved, namely DL framework, DL compiler, and DL hardware library. It then provides definitions for DL underlying library bugs and testing. Additionally, this paper summarizes the existing testing methods and tools tailored to these DL libraries separately and analyzes their effectiveness and limitations. It also discusses the existing challenges of DL library testing and outlines potential directions for future research.

Problem

• Deep Learning Library Testing
• Bug Characteristics Analysis
• Testing Methods Evaluation

Innovation

• Deep learning library testing methods
• DL framework and compiler analysis
• Bug identification and security enhancement