🤖 AI Summary
Machine learning teams face significant challenges in multi-workflow concurrent environments, including redundant intermediate data storage, inefficient cross-pipeline sharing, and high collaboration overhead. To address these issues, this paper proposes and implements a data virtualization service architecture tailored for ML workflows. The architecture adopts a service-oriented design, integrating distributed data management with dynamic metadata mapping to enable logical abstraction, on-demand loading, and unified access to heterogeneous intermediate data. Compared to conventional materialized storage approaches, it reduces storage overhead by an average of 62% (measured empirically) and substantially decreases inter-team collaboration latency. The system has been deployed in production, stably supporting six ML applications and over thirty concurrent workflows, demonstrating linear scalability. This work establishes a lightweight, elastic, and reusable data virtualization paradigm for large-scale ML infrastructure.
📝 Abstract
Nowadays, machine learning (ML) teams have multiple concurrent ML workflows for different applications. Each workflow typically involves many experiments, iterations, and collaborative activities and commonly takes months and sometimes years from initial data wrangling to model deployment. Organizationally, there is a large amount of intermediate data to be stored, processed, and maintained. emph{Data virtualization} becomes a critical technology in an infrastructure to serve ML workflows. In this paper, we present the design and implementation of a data virtualization service, focusing on its service architecture and service operations. The infrastructure currently supports six ML applications, each with more than one ML workflow. The data virtualization service allows the number of applications and workflows to grow in the coming years.