Demystifying the Silence of Correctness Bugs in PyTorch Compiler

2026-04-09

AI Summary

This work addresses silent correctness bugs in the PyTorch compiler (torch.compile), which produce erroneous model outputs without raising errors, thereby posing a critical threat to the reliability of large language models. The study presents the first systematic empirical analysis of the characteristics of such defects and introduces AlignGuard, the first targeted detection method designed specifically for this problem. AlignGuard integrates fuzz testing with large language model–guided test case mutation, augmented by a rigorous correctness validation mechanism. The approach successfully uncovered 23 previously unknown bugs, all of which have been acknowledged or fixed by the PyTorch team, including 14 classified as high-priority, substantially enhancing the compiler’s trustworthiness.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Performance optimization of AI infrastructure is key to the fast adoption of large language models (LLMs). The PyTorch compiler (torch.compile), a core optimization tool for deep learning (DL) models (including LLMs), has received due attention. However, torch.compile is prone to correctness bugs, which cause incorrect outputs of compiled DL models without triggering exceptions, crashes, or warnings. These bugs pose a serious threat to the reliability of downstream LLM applications. Data from the PyTorch community shows that 19.2% of high-priority issues are incorrect outputs of compiled DL models induced by torch.compile bugs, the second-most-common bug category (only behind program crashes at 19.57%). However, no systematic study has been conducted to specifically characterize and thereby detect these bugs. In this paper, we present the first empirical study of the correctness bugs in torch.compile, examine their characteristics, and assess the effectiveness of existing fuzzers in detecting them. Based on our findings, we propose a proof-of-concept testing technique named AlignGuard, tailored specifically for detecting correctness bugs in torch.compile. AlignGuard incorporates bug characteristics distilled from our empirical study, applying LLM-based test mutation to existing test cases for correctness bug detection. At the time of writing, AlignGuard has successfully detected 23 new correctness bugs in recent torch.compile. All these bugs have been confirmed or fixed by the PyTorch development team, and over half (14/23) of them are even marked as high-priority bugs, underscoring the usefulness of our technique.

Problem

• correctness bugs
• PyTorch compiler
• torch.compile
• silent bugs
• LLM reliability

Innovation

• correctness bugs
• torch.compile
• empirical study
• LLM-based test mutation
• AlignGuard