weights and biases

This work addresses the challenge of defect detection in deep learning libraries such as TensorFlow and PyTorch, where complex APIs often lead to subtle bugs and existing testing approaches suffer from high false-positive rates due to imprecise specifications. To overcome this limitation, the authors propose a machine learning classifier that leverages tensor shape abstraction as a precise input representation for API validity constraints. By integrating runtime feedback to automatically generate labeled training data, the method learns accurate usage patterns without relying on manual annotations. Implemented within the ACETest framework, the approach achieves over 91% classification accuracy across 183 APIs and significantly improves test pass rates—from 29% to 61%—demonstrating enhanced precision and scalability in testing deep learning libraries.

Existing automated testing tools (e.g., Pynguin) fail to recognize complex input constraints of ML library APIs (e.g., PyTorch/TensorFlow), leading to frequent early test failures and limited coverage. To address this, we propose a constraint-guided unit test generation method that jointly leverages static analysis and natural language processing to automatically extract structured constraints—including tensor dimensions, data types, and value ranges—from official documentation. We extend Pynguin to support constraint-aware test input generation. Evaluated on 165 mainstream ML modules, our approach improves code coverage by up to 63.9% over Pynguin, significantly enhancing test effectiveness and input validity. This work represents the first systematic modeling and test-driven utilization of semantic constraints inherent in deep learning library APIs.

This study presents the first systematic comparison of performance anti-patterns—i.e., recurring suboptimal coding practices harming runtime efficiency—between machine learning (ML) and non-ML Python projects. Analyzing 300 open-source GitHub repositories, we employed the RIdiom tool to detect anti-patterns and conducted empirical analysis to characterize their distribution and root causes. Results show that ML projects exhibit significantly higher prevalence of performance anti-patterns, concentrated predominantly in data preprocessing and also pervasive in model deployment stages. Crucially, the study reveals stage-specific anti-pattern distributions across the ML pipeline, moving beyond prior “black-box” analyses that treat ML projects monolithically. Key contributions are: (1) empirical validation that performance risks in ML systems are intrinsically stage-dependent; (2) demonstration that pipeline-stage-aware optimization strategies are both necessary and feasible; and (3) actionable, stage-targeted recommendations for improving data processing and model deployment efficiency.

Machine learning systems are widely deployed in sensitive domains, yet their potentially discriminatory decisions raise serious fairness concerns. Although numerous open-source fairness APIs exist for bias detection and mitigation, developers commonly face challenges—including knowledge gaps, technical barriers, and resource dependencies. This paper presents the first large-scale empirical study of fairness tool usage, analyzing 204 GitHub repositories employing 13 mainstream fairness tools. We identify two primary application objectives and 17 concrete use cases. Through systematic code review, documentation analysis, and community discussion mining, we categorize 12 representative obstacles spanning technical implementation and domain understanding. Our findings reveal a pronounced “capability–tool” gap in fairness practice—i.e., a misalignment between developers’ proficiency and tool requirements. The study provides empirically grounded insights and actionable recommendations for fairness-aware tool design, engineering education, and software development process improvement.

Contemporary machine learning research overemphasizes model performance while neglecting resource efficiency and long-term sustainability. Method: This paper systematically identifies memory/GPU-memory leak–inducing code smells in PyTorch, TensorFlow, and Keras—derived from developer community discussions and real-world code snippets—yielding 30 PyTorch-specific and 16 TensorFlow/Keras-specific anti-patterns. It proposes the first cross-framework resource leakage taxonomy, balancing generality with framework-specific adaptability, validated through a three-stage empirical process: qualitative analysis, normative refinement, and practical evaluation. Contribution/Results: The study yields 50 actionable best practices for mitigating resource leaks. It fills a critical gap in ML engineering research on resource efficiency and provides a structured, methodology-grounded coding guide for building efficient, robust, and sustainable machine learning systems.

Scientific machine learning experiments often suffer from distorted performance evaluations due to poor experimental design and inconsistent documentation. To address this, we propose a principled framework for ML experimentation tailored to scientific research, encompassing data preprocessing, model selection, cross-validation, and reporting—emphasizing reproducibility, fair comparison, and transparency. Our key contributions include two novel quantitative metrics: the Logarithmic Overfitting Ratio (LOR) and Composite Overfitting Score (COS), which jointly characterize overfitting severity and instability across cross-validation folds. Complementing these, we introduce standardized preprocessing protocols, rigorously defined strong baselines, and modular visualization templates for diagnostic analysis. Empirical evaluation demonstrates that our framework substantially enhances experimental rigor, reproducibility, and result credibility in scientific ML. It further enables robust performance assessment and cross-study comparability, providing systematic support for establishing reliable benchmarks.

This work addresses the lack of transparent, scalable, and deeply PyTorch-integrated open-source tools for post-training large language models, which hinders research iteration and deployment efficiency. We propose a native PyTorch-based, modular post-training framework centered on the principle of “hackability,” offering composable model builders, training recipes, and a distributed training stack that support diverse fine-tuning strategies and hardware configurations. While maintaining high performance and memory efficiency, the framework significantly enhances code transparency and research flexibility. Empirical evaluations demonstrate that it matches or even surpasses mainstream tools such as Axolotl and Unsloth across multiple post-training scenarios, thereby facilitating efficient and reproducible scientific exploration.

This study addresses the lack of systematic understanding regarding how project characteristics influence machine learning (ML)-specific code smells versus general Python code smells in ML projects, a gap that limits the generalizability of quality assurance strategies. Analyzing 279 open-source ML projects, the authors examine six project attributes—size, age, number of contributors, commit frequency, CI/CD adoption, and application domain—and employ CodeSmell and Pylint to detect ML-specific and general code smells, respectively. Statistical analysis reveals that ML-specific smells are significantly less prevalent than general ones; only commit frequency and application domain notably affect ML code quality. Domains such as MLOps, reinforcement learning, and computer vision exhibit distinct ML-specific smells, whereas general code smells show no significant association with project features. These findings challenge conventional technical debt assumptions and underscore the need for smell-type-specific quality strategies and domain-aware quality gates in CI/CD pipelines.

A long-standing gap exists in Python machine learning systems (MLS) between code changes and measurable software quality improvements, hindering quality-driven development practices. Method: This paper introduces the first large-scale empirical framework for MLS, proposing a context-aware taxonomy of 13 code change types and developing PyQu—a tool that integrates low-level software metrics, ML-specific indicators, and topic modeling to automatically identify quality-enhancing commits. Contribution/Results: Evaluated on 3,340 open-source projects and 3.7 million commits, PyQu achieves an average F1-score of 0.85. It discovers 41% more quality-improving changes than prior approaches and precisely identifies 61 representative modifications with statistically significant quality gains. The study fills a critical empirical void in establishing causal links between code changes and quality outcomes in MLS, delivering an interpretable, reusable methodological foundation for intelligent quality assurance.

Runtime issues—such as data drift and system changes—in dynamic production environments frequently degrade ML model performance, undermining system reliability and user trust. To address this, we conduct a multi-vocal literature review guided by Garousi et al.’s methodology, systematically synthesizing 136 academic papers and industrial grey literature—the first such comprehensive integration to bridge the research-practice gap in ML monitoring. Our analysis clarifies core motivations, key metrics, technical tools, and application contexts; identifies three recurrent challenges and critical research gaps; proposes an evolution path for reliability-oriented monitoring frameworks; and delivers actionable, practice-ready recommendations. This work provides both theoretical foundations for advancing ML monitoring research and practical guidance for building robust MLOps systems in industry. (149 words)