large-scale / distributed training engineering

Existing large-scale model training systems struggle to flexibly compose diverse parallelization strategies, often relying on manual expert tuning and lacking generality. This work proposes a programmable distributed training system that enables users to declaratively specify composite parallelism strategies—such as data, pipeline, and expert parallelism—through model annotations and scheduling directives. These specifications are compiled via a unified intermediate representation (IR) into device-level execution plans, fully decoupling strategy definition from runtime execution over a global compute-communication DAG. The system is the first to support automatic compilation of user-defined composite strategies, matching the performance of established approaches like ZeRO while significantly improving both performance and memory efficiency in complex scenarios such as DeepSeek-V3’s DualPipe.

Medical AI deployment is hindered by insufficient production readiness of machine learning (ML) training pipelines. Method: This paper presents a progressive architectural evolution path—monolithic (chaotic) → modular monolithic → microservices—using SPIRA, a voice-based pre-diagnostic system for respiratory insufficiency, as a case study. It systematically introduces continuous training (CT) and a software-quality-attribute-driven MLOps governance framework tailored to healthcare, integrating modular design, microservice decomposition, and engineered CI/CD pipelines. Contribution/Results: The approach significantly improves pipeline maintainability, fault tolerance, and scalability, enabling stable, iterative evolution of SPIRA. It establishes an “agile ML + robust software engineering” co-design paradigm, delivering a reusable methodology and practical benchmark for engineering medical AI in highly regulated environments.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

This work addresses the challenges of low communication efficiency and system complexity in large-scale AI training across geographically distributed data centers. It presents the first systematic characterization and joint optimization of three critical dimensions in “scale-across” training: parallelism strategy deployment, job scheduling, and network transport. By co-designing parallel placement, scheduling policies, and advanced networking techniques—and validating the approach through both real-world testbeds and large-scale simulations—the study achieves end-to-end global optimization of computation and communication. Experimental results demonstrate that the proposed solution improves training throughput by up to 64.62% over current production configurations and by 37.59% compared to state-of-the-art baselines.

This work addresses the current lack of open-source infrastructure capable of efficiently training and evaluating large-scale agents on complex tasks such as software engineering and computer operation. To this end, we propose a three-service decoupled architecture tailored for agent-environment interaction workloads, which separates the system into three independent services—model, agent, and environment—enabling fine-grained task scheduling, dynamic resource allocation, and unified interface communication. This design allows each component to scale independently and configure resources flexibly, significantly improving training efficiency and resource utilization. Experimental results demonstrate that the system can stably support tens of thousands of concurrent agent tasks, thereby filling a critical gap in infrastructure for large-scale agent training.

This work addresses the critical gap in engineering capacity and infrastructure within public research institutions that hinders the development of sovereign AI. Leveraging the European Alps supercomputer equipped with NVIDIA GH200 Grace Hopper Superchips, the study reports the first successful academic pretraining of Apertus, a 70-billion-parameter open-source multilingual large language model. By constructing a software-defined machine learning platform integrating large-scale distributed training, storage optimization, and high-speed interconnect stability techniques, the project overcomes key engineering bottlenecks in deploying high-performance computing systems for AI workloads. The resulting architecture provides a scalable and sustainably upgradable foundation for machine learning, thereby advancing both open science and the establishment of sovereign AI capabilities.

To address severe pipeline bubbles and limited throughput in heterogeneous large-model training, this paper proposes AdaPtis, an adaptive pipeline parallelism system. Methodologically, AdaPtis introduces (1) a generalizable pipeline performance model; (2) the first joint optimization of model partitioning, device placement, and micro-batch scheduling; and (3) a unified pipeline executor supporting diverse parallelism strategies. Experiments on representative heterogeneous hardware configurations demonstrate that AdaPtis achieves an average 1.42× speedup over Megatron-LM’s I-1F1B baseline, with peak improvements reaching 2.14×. These gains translate into significantly enhanced training efficiency and improved hardware resource utilization, without compromising model accuracy or training stability.

This work systematically investigates efficiency bottlenecks in large-scale LLM training across multi-GPU clusters (NVIDIA H100/H200, AMD MI250), focusing on the coupled effects of hardware utilization, power consumption, thermal throttling, and communication overhead. We conduct a multidimensional performance analysis of dense and sparse models using joint evaluation of tensor, pipeline, data, and expert parallelism—augmented with activation recomputation and compute-communication overlap. Key findings include: (i) scaling alone does not guarantee superior performance; smaller high-memory clusters outperform larger configurations in specific scenarios; (ii) tensor + pipeline parallelism often underutilizes interconnect bandwidth; and (iii) excessively large microbatches trigger power spikes and thermal throttling. Based on these insights, we propose parallelism strategy optimizations that jointly improve scalability and thermal stability. All experimental code is publicly released.

This work addresses the performance limitations of coding agents in real-world software engineering tasks, which stem from the scarcity of high-quality training data. To overcome this challenge, the authors propose ScaleSWE—a sandbox-based, multi-agent automated workflow that jointly orchestrates environment setup, unit test generation, and problem description synthesis to construct the first large-scale, highly diverse, and realistically complex software engineering dataset derived from six million GitHub pull requests. Requiring no human annotation, this approach substantially surpasses existing real-world datasets in both scale and fidelity. The resulting ScaleSWE dataset comprises 100,000 verified instances and is used to fine-tune the Qwen-30B-A3B-Instruct model, achieving a 64% resolution rate on SWE-Bench Verified—nearly tripling the performance of the baseline.

This work addresses the challenge of national supercomputing centers struggling to efficiently support the full lifecycle of foundation models—including pretraining, fine-tuning, and inference—by proposing a hybrid cloud-native platform that integrates diskless GPU-accelerated HPE Cray EX nodes with virtualized general-purpose infrastructure. Leveraging Kubernetes for unified orchestration, the platform bridges traditional HPC batch processing and AI-serving workflows, enabling, for the first time in a national supercomputing environment, an end-to-end “AI factory” architecture for foundation models. This approach effectively closes the paradigm gap between high-performance computing and cloud-native AI services, substantially enhancing user productivity and offering a reusable implementation blueprint for integrating end-to-end AI applications into supercomputing centers.