DEFault++: Automated Fault Detection, Categorization, and Diagnosis for Transformer Architectures

2026-04-30

AI Summary

This work addresses the susceptibility of Transformer models to silent performance degradation under component failures and the lack of fine-grained diagnostic tools. The authors propose a hierarchical diagnostic framework that first detects the presence of a fault, then classifies it into one of 12 Transformer-specific failure categories, and finally pinpoints one of 45 root causes. Key contributions include the first three-tier automated diagnosis system tailored for Transformers, integrating fault propagation graphs, prototype matching, and supervised contrastive learning to enable interpretable diagnostics; the release of DEFault-bench, a benchmark comprising 3,739 annotated samples; and DEForm, a mutation-based technique for generating high-quality training and evaluation data. Experiments show the method achieves over 0.96 AUROC in fault detection and 0.85 Macro-F1 in both classification and root cause diagnosis on DEFault-bench, significantly improving developers’ repair decision accuracy from 57.1% to 83.3%.

Abstract

Transformer models are widely deployed in critical AI applications, yet faults in their attention mechanisms, projections, and other internal components often degrade behavior silently without raising runtime errors. Existing fault diagnosis techniques often target generic deep neural networks and cannot identify which transformer component is responsible for an observed symptom. In this article, we present DEFault++, a hierarchical learning-based diagnostic technique that operates at three level of abstraction: it detects whether a fault is present, classifies it into one of 12 transformer-specific fault categories (covering both attention-internal mechanisms and surrounding architectural components), and identifies the underlying root cause from up to 45 mechanisms. To facilitate both training and evaluation, we construct DEFault-bench, a benchmark of 3,739 labeled instances obtained through systematic mutation testing. These instances are created across seven transformer models and nine downstream tasks using DEForm, a transformer-specific mutation technique we developed for this purpose. DEFault++ measures runtime behavior at the level of individual transformer components. It organizes these measurements through a Fault Propagation Graph (FPG) derived from the transformer architecture. It then produces an interpretable diagnosis using prototype matching combined with supervised contrastive learning. On DEFault-bench, DEFault++ exceeds an AUROC of 0.96 for detection and a Macro-F1 of 0.85 for both categorization and root-cause diagnosis on encoder and decoder architectures. In a developer study with 21 practitioners, the accuracy of choosing correct repair actions increased from 57.1% without support to 83.3% when using DEFault++.

Problem

• Transformer
• fault detection
• fault diagnosis
• attention mechanism
• model reliability

Innovation

• Transformer fault diagnosis
• Fault Propagation Graph
• supervised contrastive learning
• mutation testing
• prototype matching