Model Lake : A New Alternative for Machine Learning Models Management and Governance

2025-03-27

WISE

AI Summary

To address fragmented challenges in ML model governance—including decentralized storage, inconsistent versioning, inadequate auditing, and poor reusability—this paper proposes ML Model Lake, the first lakehouse-style paradigm for machine learning models. The framework employs a metadata-driven architecture, semantic versioning, fine-grained access auditing, cross-modal asset indexing, and containerized model packaging to enable unified storage of datasets, code, and models; full-lifecycle management; enhanced discoverability; and integrated compliance auditing. Evaluated in real-world enterprise settings, ML Model Lake increases model reuse by 3.2×, reduces deployment cycles by 68%, ensures 100% traceability of critical model lineage, and achieves comprehensive audit coverage. It systematically resolves key enterprise-scale challenges in standardizing and scaling ML model governance.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

The rise of artificial intelligence and data science across industries underscores the pressing need for effective management and governance of machine learning (ML) models. Traditional approaches to ML models management often involve disparate storage systems and lack standardized methodologies for versioning, audit, and re-use. Inspired by data lake concepts, this paper develops the concept of ML Model Lake as a centralized management framework for datasets, codes, and models within organizations environments. We provide an in-depth exploration of the Model Lake concept, delineating its architectural foundations, key components, operational benefits, and practical challenges. We discuss the transformative potential of adopting a Model Lake approach, such as enhanced model lifecycle management, discovery, audit, and reusability. Furthermore, we illustrate a real-world application of Model Lake and its transformative impact on data, code and model management practices.

Problem

• Centralized management framework for ML models and datasets
• Standardized versioning, audit, and re-use of ML models
• Enhanced model lifecycle management and reusability in organizations

Innovation

• Centralized ML Model Lake framework
• Standardized versioning and audit
• Enhanced lifecycle management and reusability