XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries

2025-08-17

AI Summary

Existing deep learning library fuzzing approaches primarily rely on behavioral discrepancies across hardware backends (e.g., CPU vs. GPU) to detect bugs, making them ineffective at uncovering subtle functional inconsistencies that manifest only across frameworks—despite correct behavior within each individual framework. This work proposes XAMT, the first fuzzing methodology targeting cross-framework API functional equivalence. XAMT identifies semantically equivalent APIs across PyTorch, TensorFlow, and other frameworks via name, documentation, and parameter-structure similarity; it then applies input alignment and variance-guided differential testing to expose consistency violations. Departing from hardware-difference-driven paradigms, XAMT constructs a benchmark of 238 equivalent interface pairs (covering 839 APIs), detects 17 previously unknown defects, and achieves 12 confirmed fixes. This significantly extends the scope and effectiveness of deep learning library testing along the cross-framework functional consistency dimension.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Learning & Optimization for CV
• Reasoning under Uncertainty: Probabilistic Programming

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Deep learning powers critical applications such as autonomous driving, healthcare, and finance, where the correctness of underlying libraries is essential. Bugs in widely used deep learning APIs can propagate to downstream systems, causing serious consequences. While existing fuzzing techniques detect bugs through intra-framework testing across hardware backends (CPU vs. GPU), they may miss bugs that manifest identically across backends and thus escape detection under these strategies. To address this problem, we propose XAMT, a cross-framework fuzzing method that tests deep learning libraries by matching and comparing functionally equivalent APIs across different frameworks. XAMT matches APIs using similarity-based rules based on names, descriptions, and parameter structures. It then aligns inputs and applies variance-guided differential testing to detect bugs. We evaluated XAMT on five popular frameworks, including PyTorch, TensorFlow, Keras, Chainer, and JAX. XAMT matched 839 APIs and identified 238 matched API groups, and detected 17 bugs, 12 of which have been confirmed. Our results show that XAMT uncovers bugs undetectable by intra-framework testing, especially those that manifest consistently across backends. XAMT offers a complementary approach to existing methods and offers a new perspective on the testing of deep learning libraries.

Problem

• Detects bugs in deep learning APIs across frameworks
• Identifies consistent bugs missed by intra-framework testing
• Matches equivalent APIs for differential testing

Innovation

• Cross-framework API matching for testing
• Similarity-based API matching rules
• Variance-guided differential testing method