Provenance Tracking in Large-Scale Machine Learning Systems

2025-07-01

AI Summary

In large-scale machine learning systems, jointly optimizing computational efficiency, execution time, accuracy, and energy consumption remains challenging, while training workflows often lack reproducibility and accountability. To address these issues, this paper introduces yProv4ML—a lightweight, open-source library enabling fine-grained provenance capture. Strictly adhering to the W3C PROV and ProvML standards, yProv4ML efficiently collects end-to-end provenance metadata—including data sources, contextual information, and transformation steps—in compact JSON format. Its modular, plugin-based architecture enables seamless integration with heterogeneous monitoring tools and workflow management systems. Experimental evaluation demonstrates that yProv4ML accurately pinpoints performance bottlenecks, quantifies resource utilization patterns (e.g., CPU, memory, energy), and supports green AI assessment. Crucially, it enhances observability and engineering controllability of ML development pipelines without compromising interpretability or reproducibility.

Technology Category

• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

As the demand for large scale AI models continues to grow, the optimization of their training to balance computational efficiency, execution time, accuracy and energy consumption represents a critical multidimensional challenge. Achieving this balance requires not only innovative algorithmic techniques and hardware architectures but also comprehensive tools for monitoring, analyzing, and understanding the underlying processes involved in model training and deployment. Provenance data information about the origins, context, and transformations of data and processes has become a key component in this pursuit. By leveraging provenance, researchers and engineers can gain insights into resource usage patterns, identify inefficiencies, and ensure reproducibility and accountability in AI development workflows. For this reason, the question of how distributed resources can be optimally utilized to scale large AI models in an energy efficient manner is a fundamental one. To support this effort, we introduce the yProv4ML library, a tool designed to collect provenance data in JSON format, compliant with the W3C PROV and ProvML standards. yProv4ML focuses on flexibility and extensibility, and enables users to integrate additional data collection tools via plugins. The library is fully integrated with the yProv framework, allowing for higher level pairing in tasks run also through workflow management systems.

Problem

• Optimize training of large AI models for efficiency and accuracy
• Track provenance data to monitor and analyze model training
• Develop tools for scalable, energy-efficient AI resource utilization

Innovation

• Uses provenance data for AI model tracking
• Introduces yProv4ML library for JSON data
• Integrates plugins for flexible data collection