🤖 AI Summary
Existing deep learning–based fault diagnosis methods suffer significant performance degradation on unseen programs, primarily due to a mismatch between conventional evaluation strategies and real-world deployment scenarios. This work introduces DynFault, a novel dataset comprising 38 real-world deep learning programs and 5,542 fault-injection traces, which for the first time systematically reveals and quantifies the performance gap—measured as a 0.190 drop in accuracy—between intra-program cross-validation and leave-one-program-out evaluation. The study identifies program-level feature structure as a key generalization bottleneck: curvature-based features prove effective for instability detection on unseen programs, whereas optimizer- and activation-related features exhibit predictive power only within seen programs, highlighting the heterogeneous generalizability of runtime features across programs.
📝 Abstract
Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task. Techniques for diagnosing such failures are commonly assessed using within-program cross-validation, which may be inadequate for deployment settings involving previously unseen programs. It is therefore necessary to assess how performance differs across these settings and to identify the causes of any performance gap in established fault diagnosis techniques for DL. We investigate this gap using DynFault, a corpus of 5,542 fault-injected training traces from 38 real-world DL programs. We found a gap of 0.190 in balanced accuracy for existing fault diagnosis techniques between within-program evaluation and holding out whole programs. We also found the gap comes from program-level structure in the features, which led us to examine two runtime feature sets, curvature features and optimizer features, and their behavior on unseen programs. We found that curvature features are useful for instability detection on unseen programs, while optimizer and activation features help only on programs seen during training.