kubeflow

To address memory reliability challenges in large-scale AI training on Kubernetes—including OOM kills, over-allocation, memory leaks, and ephemeral storage exhaustion—this paper proposes the first memory governance framework tailored for ML workloads. Methodologically, it introduces a GPU-aware memory quota policy that jointly constrains GPU memory and system memory; designs a dynamic, cgroup v2–based elastic reclaim mechanism for ephemeral storage; and integrates a Prometheus/Grafana observability stack with a custom Eviction Advisor. Experiments on real-world, thousand-GPU distributed training clusters demonstrate a 92% reduction in OOM incidents, a 37% increase in GPU memory utilization, and SLA compliance exceeding 99.5% for training jobs. The core contribution lies in unifying memory QoS enforcement, GPU–system memory coupling modeling, and elastic ephemeral storage management within Kubernetes’ native scheduling architecture—marking the first such holistic approach.

Machine learning teams face significant challenges in multi-workflow concurrent environments, including redundant intermediate data storage, inefficient cross-pipeline sharing, and high collaboration overhead. To address these issues, this paper proposes and implements a data virtualization service architecture tailored for ML workflows. The architecture adopts a service-oriented design, integrating distributed data management with dynamic metadata mapping to enable logical abstraction, on-demand loading, and unified access to heterogeneous intermediate data. Compared to conventional materialized storage approaches, it reduces storage overhead by an average of 62% (measured empirically) and substantially decreases inter-team collaboration latency. The system has been deployed in production, stably supporting six ML applications and over thirty concurrent workflows, demonstrating linear scalability. This work establishes a lightweight, elastic, and reusable data virtualization paradigm for large-scale ML infrastructure.

To address inefficiencies in model incremental updates, unfair policy evaluation, and high retraining costs under continual data growth, this paper proposes an end-to-end adaptive machine learning platform. Methodologically: (1) it introduces a declarative domain-specific language (DSL) to uniformly model data selection strategies (e.g., coreset, uncertainty sampling) and trigger policies (e.g., drift-aware scheduling); (2) it establishes the first composite model evaluation framework enabling fair, cross-policy comparison; and (3) it implements a co-optimization mechanism integrating sample-level fine-grained data selection with high-throughput training. Contributions include an open-source, extensible system architecture, a standardized benchmark ecosystem, and abstracted ML pipeline interfaces. Experiments demonstrate significant improvements in training throughput and substantial reductions in retraining overhead—while preserving model accuracy—and enable reproducible analysis across diverse strategy combinations.

To address the challenges of GPU accelerator resource scheduling and weak cross-site collaboration for fundamental scientific research (e.g., high-energy physics) in federated cloud environments, this paper proposes a heterogeneous federated Kubernetes architecture leveraging Virtual Kubelets and interLink. The architecture enables fine-grained GPU virtualization, dynamic orchestration, and cross-domain sharing across multi-cloud infrastructures—achieving unified infrastructure management while preserving scientific use-case diversity. Built on cloud-native containerization, the platform integrates GPU-aware scheduling optimizations and federated network coordination mechanisms. Experimental evaluation demonstrates that the AI development platform supports concurrent GPU-intensive workloads from multiple research teams, improves GPU resource utilization by 40%, and reduces cross-datacenter workflow scheduling latency by 60%. These results significantly enhance the agility and scalability of distributed, data-driven scientific analysis in federated cloud settings.

This work addresses the practical challenges of deploying machine learning models in real-world settings, where heterogeneous data protocols, non-standard formats, and infrastructure constraints often necessitate redundant construction of integration pipelines. To overcome these issues, we propose SMOCS—a containerized, streaming ML system built on Apache Kafka—that decouples infrastructure from application logic through layered abstraction and employs a three-threaded agent architecture to separate data ingestion, online training, and real-time inference. The framework enables configuration-driven, no-code deployment, offering platform independence, fault isolation, and horizontal scalability, thereby significantly lowering the barrier to entry for domain experts. SMOCS has been open-sourced on the Jefferson Lab GitHub repository and demonstrates both continuous online learning capability and strong engineering practicality.

This work addresses customs clearance delays in global trade caused by ambiguous product descriptions and frequent updates to Harmonized System (HS) codes. To tackle this challenge, the authors propose a serverless MLOps framework that leverages event-driven pipelines and managed services to enable end-to-end, model-agnostic machine learning lifecycle management. The architecture supports automatic scaling, reproducible training, auditable deployment, and automated A/B testing, ensuring secure and seamless model transitions. By integrating custom text embeddings with models such as Text-CNN, the system achieves 98% accuracy on real-world HS code prediction tasks, meeting stringent service-level agreement (SLA) requirements. This approach significantly reduces long-term operational costs and establishes an efficient, cost-effective, and reproducible deployment paradigm for industrial-scale machine learning systems.

This study addresses the lack of systematic comparisons among mainstream MLOps tools, which hinders developers’ ability to select appropriate solutions for their specific needs. For the first time, it presents a multi-dimensional empirical evaluation of MLflow, Metaflow, Apache Airflow, and Kubeflow Pipelines under a unified experimental setup, using two representative tasks: MNIST image classification and IMDB sentiment analysis with BERT. The assessment spans six key criteria—installation ease, configuration flexibility, interoperability, code intrusiveness, result interpretability, and documentation quality—and incorporates a weighted scoring mechanism. By establishing a balanced evaluation framework that integrates both quantitative and qualitative insights, this work delivers a clear and practical guide for selecting MLOps tools tailored to diverse application scenarios.

This work addresses the challenge faced by scientific and machine learning researchers in efficiently leveraging multi-cloud resources for complex computations due to a lack of cloud computing expertise. To bridge this gap, we propose a workflow-centric multi-cloud platform that encapsulates low-level operations—such as environment setup, instance selection, data migration, and distributed execution—into high-level, declarative interfaces through reusable, expert-designed workflows. Users need only specify their computational intent, and the platform automatically handles cross-cloud scheduling and optimization. The system has successfully supported glaciological simulation applications such as Icepack and PISM, significantly simplifying deployment without requiring HPC or cloud specialization, while enabling efficient cost–performance trade-off analysis and scalability exploration to accelerate scientific discovery.

Existing LLM-based Kubernetes diagnostic systems cannot learn from operational experience, operating on static knowledge bases without improving from past resolutions. We present MetaKube, an experience-aware LLM framework through three synergistic innovations: (1) an Episodic Pattern Memory Network (EPMN) that abstracts diagnostic patterns from historical resolutions and provides confidence-calibrated retrieval for both rapid pattern matching and guided causal exploration, (2) a meta-cognitive controller that dynamically routes between intuitive and analytical pathways based on problem familiarity, optimizing the trade-off between speed and depth, and (3) KubeLLM, a locally-deployable 8B model enhanced through domain-specific post-training on our 7,000-sample Kubernetes Fault Resolution Dataset. Evaluation on 1,873 real-world scenarios demonstrates MetaKube transforms Qwen3-8B from 50.9 to 90.5 points, approaching GPT-4.1 performance while ensuring complete data privacy. EPMN contributes 15.3% improvement through experiential learning, with continuous learning experiments showing progressive gains as the system accumulates operational knowledge. The source code and related resources are available at https://github.com/MetaKube-LLM-for-Kubernetes-Diagnosis/MetaKube.

This work addresses the inefficiency of the current Python machine learning ecosystem in supporting large-scale, highly concurrent machine learning pipeline searches driven by large language model (LLM) agents. To overcome this limitation, we propose the first system architecture specifically designed for agent-driven ML workloads, which decouples the agent’s planning and reasoning from pipeline execution to enable batch compilation and efficient scheduling. Our system introduces pipeline graph compilation, batched execution optimization, and a high-performance Rust runtime, while seamlessly integrating with mainstream Python libraries and supporting heterogeneous backends including CPUs and GPUs. Experimental results demonstrate that our approach achieves up to a 16.6× speedup on large-scale agent-driven ML pipeline search tasks.