model training

To address memory reliability challenges in large-scale AI training on Kubernetes—including OOM kills, over-allocation, memory leaks, and ephemeral storage exhaustion—this paper proposes the first memory governance framework tailored for ML workloads. Methodologically, it introduces a GPU-aware memory quota policy that jointly constrains GPU memory and system memory; designs a dynamic, cgroup v2–based elastic reclaim mechanism for ephemeral storage; and integrates a Prometheus/Grafana observability stack with a custom Eviction Advisor. Experiments on real-world, thousand-GPU distributed training clusters demonstrate a 92% reduction in OOM incidents, a 37% increase in GPU memory utilization, and SLA compliance exceeding 99.5% for training jobs. The core contribution lies in unifying memory QoS enforcement, GPU–system memory coupling modeling, and elastic ephemeral storage management within Kubernetes’ native scheduling architecture—marking the first such holistic approach.

This work addresses the inefficiency in notebook-based distributed workflows, where minor modifications often trigger full re-execution, severely hindering iterative development and reproducibility. To overcome this limitation, the authors propose NBRewind, a system that, for the first time, enables fine-grained incremental execution and cross-platform portability while preserving reproducibility. NBRewind integrates a dual-kernel architecture—comprising auditing and replay components—with cell-level incremental checkpoints and inter-cell dataflow analysis. It further leverages standardized notebook packaging to facilitate efficient partial re-execution. Evaluation in real-world high-performance computing (HPC) scenarios demonstrates that NBRewind incurs minimal overhead for incremental checkpointing and substantially improves both execution efficiency and cross-site reproducibility.

Medical AI deployment is hindered by insufficient production readiness of machine learning (ML) training pipelines. Method: This paper presents a progressive architectural evolution path—monolithic (chaotic) → modular monolithic → microservices—using SPIRA, a voice-based pre-diagnostic system for respiratory insufficiency, as a case study. It systematically introduces continuous training (CT) and a software-quality-attribute-driven MLOps governance framework tailored to healthcare, integrating modular design, microservice decomposition, and engineered CI/CD pipelines. Contribution/Results: The approach significantly improves pipeline maintainability, fault tolerance, and scalability, enabling stable, iterative evolution of SPIRA. It establishes an “agile ML + robust software engineering” co-design paradigm, delivering a reusable methodology and practical benchmark for engineering medical AI in highly regulated environments.

This work addresses the limitation of existing distributed data pipeline systems, which require users to explicitly define complete workflow graphs, by proposing a unified planning and scheduling framework that automatically constructs end-to-end persistent pipelines from implicit goal declarations alone. The approach introduces, for the first time, a numeric-domain-independent planner into the context of persistent scheduling, integrating workflow and resource graph modeling, numeric planning, and network interface scheduling to achieve full automation. Experimental results demonstrate the feasibility and scalability of the method: under a single-machine constraint of one hour of CPU time and 30 GB of memory, the system successfully scheduled a linear pipeline spanning eight sites and comprising fourteen components.

In large-scale DNN distributed training, checkpointing is tightly coupled with model parallelism strategies and hardware topology, severely limiting fault tolerance and elastic scalability. To address this, we propose the “distributed storage, unified loading” paradigm: during saving, model parameters are stored in a distributed representation aligned with the current parallel configuration; during restoration, they are uniformly reconstructed into a logically consistent parameter view. We design a universal checkpoint format—incorporating merged parameter representations and mapping metadata—a Universal Checkpoint Language (UCL), and an on-demand state reconstruction mechanism, achieving, for the first time, full decoupling of checkpointing from parallel configurations. Evaluated on LLaMA, Bloom, and other mainstream large models under diverse parallelism paradigms—including tensor parallelism (TP), pipeline parallelism (PP), data parallelism (DP), and context parallelism (CP)—our approach reduces post-failure recovery time by 12–28% on average, significantly enhancing cross-configuration portability and system robustness.

Existing large-scale model training systems struggle to flexibly compose diverse parallelization strategies, often relying on manual expert tuning and lacking generality. This work proposes a programmable distributed training system that enables users to declaratively specify composite parallelism strategies—such as data, pipeline, and expert parallelism—through model annotations and scheduling directives. These specifications are compiled via a unified intermediate representation (IR) into device-level execution plans, fully decoupling strategy definition from runtime execution over a global compute-communication DAG. The system is the first to support automatic compilation of user-defined composite strategies, matching the performance of established approaches like ZeRO while significantly improving both performance and memory efficiency in complex scenarios such as DeepSeek-V3’s DualPipe.

This work proposes a finite state machine–based voice-guided human-robot collaborative workflow orchestration framework to address the challenges of scaling expert knowledge in industrial settings and the degradation of operational quality caused by variability among personnel and conversational interactions. By integrating speech-based intent understanding under explicit state constraints with modular workflows, the approach establishes an interpretable, reproducible, and cognitively lightweight collaboration paradigm. It also provides unified coordination of heterogeneous resources, including GUI-based software and collaborative robots. Evaluated in an industrial pilot involving turbine blade inspection and repair preparation, the system significantly reduces end-to-end process time while ensuring high repeatability and operational consistency.

Automated construction of executable ELT pipelines faces significant challenges, including ambiguous user intent, unreliable tool generation, and a lack of execution guarantees in outputs. This work proposes kRAIG, a natural language–driven AI agent that leverages a novel ReQuesAct interaction framework to explicitly clarify user intent. By integrating retrieval-augmented component composition with a multi-stage LLM-based verification mechanism, kRAIG automatically translates high-level instructions into production-grade Kubeflow pipelines. Experimental results demonstrate that kRAIG achieves a threefold improvement in data extraction and loading success rates and a 25% increase in transformation accuracy compared to existing approaches, substantially enhancing the reliability, executability, and practicality of automated data engineering pipelines.

Existing large model training data pipelines struggle to simultaneously ensure batch semantics, fault isolation, and consistency. This work proposes an agent-free, object-storage-native training data plane featuring three core innovations: a Transactional Global Batch (TGB) abstraction that guarantees training consistency, a storage-layer-embedded garbage collection mechanism aligning producer states with distributed checkpoints, and a communication-free Decentralized Adaptive Commit (DAC) algorithm. Leveraging lakehouse ACID semantics in object storage and distributed checkpointing, the system achieves higher throughput than colocated loaders and Kafka, lower read latency, and full fault isolation across 64-GPU multimodal pretraining and supervised fine-tuning (SFT) workloads.

This work addresses the practical challenges of deploying machine learning models in real-world settings, where heterogeneous data protocols, non-standard formats, and infrastructure constraints often necessitate redundant construction of integration pipelines. To overcome these issues, we propose SMOCS—a containerized, streaming ML system built on Apache Kafka—that decouples infrastructure from application logic through layered abstraction and employs a three-threaded agent architecture to separate data ingestion, online training, and real-time inference. The framework enables configuration-driven, no-code deployment, offering platform independence, fault isolation, and horizontal scalability, thereby significantly lowering the barrier to entry for domain experts. SMOCS has been open-sourced on the Jefferson Lab GitHub repository and demonstrates both continuous online learning capability and strong engineering practicality.