Rethinking Issue Resolution for AI/ML Systems

2026-07-16

AI Summary

Existing bug-fixing frameworks struggle to address the unique challenges posed by AI/ML systems, including non-deterministic behavior, experiment-driven workflows, and the need for coordinated changes across multiple artifacts. Through a qualitative analysis of 100 issue reports and pull requests from TensorFlow, scikit-learn, MLflow, and AutoGPT, this study systematically uncovers core characteristics of AI/ML debugging and repair—namely cross-phase activities, iterative validation, and multi-artifact coordination. The research identifies key obstacles such as reproducibility issues, behavioral non-determinism, and artifact misalignment. Building on these findings, the paper articulates a vision for a tailored bug-fixing framework specifically designed for AI/ML systems, offering an empirical foundation to guide future toolchain development and research in this emerging domain.

Abstract

We advocate for AI/ML issue resolution frameworks tailored to maintenance workflows and the nature of modern AI/ML systems. Existing issue resolution frameworks largely emerged for traditional software maintenance practices and do not explicitly account for characteristics common in AI/ML systems, such as stochastic behavior, experimentation-driven workflows, and heterogeneous artifacts beyond source code. To identify the unique characteristics of issue resolution in AI/ML systems and motivate the need for tailored frameworks, we conducted a qualitative study of issue resolution workflows documented in 100 issue reports and pull requests across four widely used AI/ML systems: TensorFlow, scikit-learn, MLflow, and AutoGPT. Our findings suggest that issue resolution in AI/ML systems involves: recurring AI/ML-related activities that span multiple resolution stages; iterative experimentation and adaptive verification; and coordinated changes across artifacts such as datasets, prompts, and model configurations. We also observed challenges related to reproducibility, nondeterministic behavior, and artifact coordination. Building on these findings, we present a vision for AI/ML issue resolution frameworks and discuss research directions and tooling support needed to realize this vision.

Problem

• AI/ML systems
• issue resolution
• software maintenance
• stochastic behavior
• heterogeneous artifacts

Innovation

• AI/ML issue resolution
• experimentation-driven workflow
• heterogeneous artifacts
• adaptive verification
• artifact coordination