disaggregated serving

To address the challenges of cross-platform orchestration and fragmented resource scheduling in hybrid HPC–ML workflows, this paper proposes a service-oriented, scalable runtime architecture. Building upon the RADICAL-Pilot framework, we introduce the first service-oriented execution model enabling dynamic, multi-granularity, low-overhead coordination of heterogeneous HPC and ML tasks. Our approach unifies resource abstraction across platforms, implements distributed task scheduling, and jointly orchestrates AI and HPC workloads—thereby enabling seamless coupling and coordinated scheduling between on-premises exascale supercomputers and cloud environments. Experimental evaluation on an exascale prototype system demonstrates concurrent deployment of multiple ML models with runtime overhead under 2%. The architecture successfully supports three representative data-driven scientific applications, effectively overcoming the traditional siloing of HPC and ML workflows.

To address fragmented challenges in ML model governance—including decentralized storage, inconsistent versioning, inadequate auditing, and poor reusability—this paper proposes ML Model Lake, the first lakehouse-style paradigm for machine learning models. The framework employs a metadata-driven architecture, semantic versioning, fine-grained access auditing, cross-modal asset indexing, and containerized model packaging to enable unified storage of datasets, code, and models; full-lifecycle management; enhanced discoverability; and integrated compliance auditing. Evaluated in real-world enterprise settings, ML Model Lake increases model reuse by 3.2×, reduces deployment cycles by 68%, ensures 100% traceability of critical model lineage, and achieves comprehensive audit coverage. It systematically resolves key enterprise-scale challenges in standardizing and scaling ML model governance.

To address the scalability lag and poor burst-load responsiveness in serverless LLM inference—caused by high model loading overhead—this paper proposes λPipe, a distributed inference architecture integrating RDMA-accelerated multicast with an execute-while-load mechanism. Its core contributions are threefold: (1) an adaptive multi-node pipelined scheduler enabling hierarchical model management across GPU and host memory, along with real-time collaborative inference; (2) heterogeneous memory–aware dynamic model loading; and (3) deep optimizations of the serverless runtime. Evaluated on realistic LLM inference traces, λPipe reduces tail latency by up to 5× and lowers service cost by 31.3% compared to baseline approaches, significantly improving both responsiveness under bursty workloads and resource efficiency.

In large language model (LLM) serving, autoscaling faces a fundamental trade-off between high scaling latency and substantial parameter caching overhead: conventional approaches rely on local parameter caches, causing service interruption during model loading and suffering from data-plane bottlenecks in cross-host scaling. This paper proposes a local-cache-free, millisecond-scale real-time autoscaling framework. It introduces the first O(1)-complexity network-based direct parameter transmission mechanism, enabling zero-copy parameter loading over GPU-to-GPU high-speed interconnects. We design a layer-granularity dynamic collaborative execution architecture supporting fine-grained load migration and multicast-based parameter distribution. Integrated cooperative inference scheduling eliminates cold-start delays. Experiments show up to 86% reduction in tail latency, achieving performance close to the ideal full-host, full-parameter caching configuration—while completely eliminating local parameter storage overhead.

To address challenges in high-performance computing (HPC) environments—including heterogeneous LLM deployment, inflexible resource scheduling, and significant performance volatility under multi-model concurrent inference—this paper proposes a scalable LLM inference engine architecture built atop SLURM. The architecture integrates containerized microservices with dynamic resource orchestration, enabling fine-grained, coordinated allocation of CPU, GPU, and memory resources, and provides unified access via RESTful APIs to support both batch and interactive inference workloads. A novel multi-step “tribunal” refinement workflow is introduced to enhance fault tolerance and operational flexibility. Experiments on Llama-series models across multi-node HPC clusters demonstrate sub-50 ms latency and 128 concurrent requests for smaller models (e.g., Llama-3-8B), and stable dual-concurrent execution for large models (e.g., Llama-3-70B), with low scheduling overhead and strong horizontal scalability. The system has been successfully deployed in production applications, including retrieval-augmented generation chatbots.

This work addresses the significant degradation in inference throughput caused by GPU memory constraints when concurrently deploying multiple large language models on shared heterogeneous hardware, where resource scheduling, model offloading, and preemption become critical bottlenecks. Through empirical methodologies—including cross-platform performance profiling, layer-wise offloading experiments, and fine-grained decomposition of preemption overhead—the study systematically uncovers, for the first time, the nonlinear relationship between offloading and throughput decline. It further identifies model state reloading as the primary source of preemption overhead. The findings reveal that smaller models are more sensitive to reduced GPU residency, and that such overhead is jointly influenced by model architecture and hardware characteristics. These insights motivate a scheduler design that integrates model-specific sensitivity with data migration costs, offering crucial guidance for building efficient multi-model serving systems.

This work addresses the scheduling and resource management challenges in large model inference caused by high GPU memory consumption in chain-of-thought workloads. It formally defines the "server chain composition" problem for the first time, proves its NP-hardness, and introduces a scalable algorithm with provable performance guarantees. The proposed approach integrates pipeline parallelism, block placement strategies, cache allocation mechanisms, and advanced load balancing techniques to jointly optimize server chain construction and resource scheduling. Experimental results demonstrate that the method significantly reduces response latency in distributed large language model serving systems and outperforms state-of-the-art approaches.

Current large language model inference systems lack the capability to dynamically adjust model parallelism topologies at runtime, necessitating service restarts under varying workloads and causing multi-minute disruptions, loss of KV cache, and substantial recomputation overhead. This work proposes ReMP, the first framework enabling online elastic reconfiguration of combined tensor and pipeline parallelism. By decoupling topology from execution state, designing a two-dimensional KV cache migration mechanism, and orchestrating an end-to-end reconfiguration pipeline, ReMP reduces topology switching latency to 1–7 seconds across 7B–70B models—orders of magnitude faster than restarting. This significantly improves time-to-first-token (TTFT), time per output token (TPOT), and throughput under dynamic workloads.

This work addresses customs clearance delays in global trade caused by ambiguous product descriptions and frequent updates to Harmonized System (HS) codes. To tackle this challenge, the authors propose a serverless MLOps framework that leverages event-driven pipelines and managed services to enable end-to-end, model-agnostic machine learning lifecycle management. The architecture supports automatic scaling, reproducible training, auditable deployment, and automated A/B testing, ensuring secure and seamless model transitions. By integrating custom text embeddings with models such as Text-CNN, the system achieves 98% accuracy on real-world HS code prediction tasks, meeting stringent service-level agreement (SLA) requirements. This approach significantly reduces long-term operational costs and establishes an efficient, cost-effective, and reproducible deployment paradigm for industrial-scale machine learning systems.

Existing simulators struggle to jointly model the runtime software-hardware interactions in heterogeneous hardware environments and disaggregated large language model (LLM) serving architectures, limiting systematic evaluation of performance, memory, and power consumption. This work proposes the first unified system-level simulator that integrates service scheduling and hardware behavior within a shared runtime loop, explicitly capturing dynamic interactions—including batching, routing, offloading, and energy efficiency—under heterogeneous accelerators, near-memory computing, and disaggregated resource deployment. The framework employs a profiling-driven modeling approach, enabling scalable integration of emerging hardware and establishing an evaluation pathway for co-design between hardware and serving systems. Experimental results demonstrate that simulations complete in approximately 10 minutes with an average error of only 0.97% on key metrics, achieving both high fidelity and practical usability.