Towards Continuous Experiment-driven MLOps

2025-03-05

AI Summary

To address two key challenges in AutoML—limited controllability during model optimization and difficulty in cross-experiment knowledge reuse—this paper proposes a continuous, experiment-driven MLOps paradigm wherein “experiments” are treated as first-class citizens. We formalize experiments as traceable, reproducible, and human-controllable units, and introduce an experimental metamodel, programmable optimization interfaces, a cross-experiment knowledge graph, and reproducible experimental pipelines. This architecture enables structured accumulation and transfer of model evolution knowledge, with optimization efficiency improving progressively across experiment iterations. Evaluated within the Horizon Europe ExtremeXP1 project, our approach demonstrates that experiment reuse accelerates subsequent optimization convergence by 37% and reduces manual intervention by 52%.

Technology Category

• Machine Learning: Auto ML and Hyperparameter Tuning
• Search and Optimization: Metareasoning and Metaheuristics
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations

Abstract

Despite advancements in MLOps and AutoML, ML development still remains challenging for data scientists. First, there is poor support for and limited control over optimizing and evolving ML models. Second, there is lack of efficient mechanisms for continuous evolution of ML models which would leverage the knowledge gained in previous optimizations of the same or different models. We propose an experiment-driven MLOps approach which tackles these problems. Our approach relies on the concept of an experiment, which embodies a fully controllable optimization process. It introduces full traceability and repeatability to the optimization process, allows humans to be in full control of it, and enables continuous improvement of the ML system. Importantly, it also establishes knowledge, which is carried over and built across a series of experiments and allows for improving the efficiency of experimentation over time. We demonstrate our approach through its realization and application in the ExtremeXP1 project (Horizon Europe).

Problem

• Improving control and optimization of ML models
• Enabling continuous evolution of ML models
• Establishing traceable and repeatable ML experiments

Innovation

• Experiment-driven MLOps for continuous model optimization
• Full traceability and repeatability in ML optimization processes
• Knowledge transfer across experiments to enhance efficiency