yProv4ML: Effortless Provenance Tracking for Machine Learning Systems

2025-07-01

AI Summary

Current large language model (LLM) development suffers from opaque training processes, empirically chosen hyperparameters (e.g., number of epochs), and poor experimental reproducibility; existing tools (e.g., MLflow) lack systematic support for data lineage. To address these issues, we propose a lightweight, framework-agnostic machine learning lineage framework built upon the standard PROV-JSON specification. It integrates seamlessly with mainstream training frameworks via automated instrumentation hooks—requiring no modifications to user code—thereby fully capturing hyperparameter configurations, data provenance, and execution traces. Our key contribution is the first deep integration of a standardized lineage model into the LLM training pipeline, enabling end-to-end, interoperable, and verifiable experiment tracking. This significantly enhances transparency in model development and strengthens scientific reproducibility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The rapid growth of interest in large language models (LLMs) reflects their potential for flexibility and generalization, and attracted the attention of a diverse range of researchers. However, the advent of these techniques has also brought to light the lack of transparency and rigor with which development is pursued. In particular, the inability to determine the number of epochs and other hyperparameters in advance presents challenges in identifying the best model. To address this challenge, machine learning frameworks such as MLFlow can automate the collection of this type of information. However, these tools capture data using proprietary formats and pose little attention to lineage. This paper proposes yProv4ML, a framework to capture provenance information generated during machine learning processes in PROV-JSON format, with minimal code modifications.

Problem

• Tracks provenance in ML systems transparently
• Addresses lack of hyperparameter transparency in LLMs
• Captures ML process data in standard PROV-JSON format

Innovation

• Automates provenance tracking for ML systems
• Uses PROV-JSON format for data capture
• Requires minimal code modifications