C2CServe: Leveraging NVLink-C2C for Elastic Serverless LLM Serving on MIG

2026-05-19

AI Summary

This work addresses the low GPU memory utilization under dedicated allocation, cold-start overheads in time-sharing, and memory capacity constraints of spatial sharing (e.g., MIG) in large language model (LLM) serving. The authors propose a serverless LLM serving architecture leveraging NVLink-C2C interconnects, which uniquely stores model weights primarily in CPU memory and streams them on-demand into MIG instances, enabling request-granularity elastic model switching. Key innovations include HybridGEMM—a heterogeneous memory-aware compute kernel—alongside a hierarchical scheduler and an online feedback control mechanism that jointly optimize C2C bandwidth contention and memory access patterns. Evaluated on GH200 systems, the approach reduces cold-start latency by 7.1× for dense models and 4.6× for mixture-of-experts (MoE) models, while maintaining over 95% compliance with TTFT and TPOT SLOs even under C2C bandwidth contention.

Abstract

Modern LLM serving is increasingly serverless in shape: large model catalogs, long-tail invocations, and multi-tenant demand. Existing GPU serving systems face a tradeoff: dedicated-GPU allocation wastes scarce HBM under sparse traffic, while GPU time sharing places model initialization and weight loading on the cold-start path. Spatial GPU sharing such as multi-instance GPU (MIG) provides isolation and accounting, but each slice has too little HBM for modern LLM weights. We observe that high-bandwidth CPU--GPU interconnects, such as NVLink-C2C (C2C) in NVIDIA GH200 and GB200 Superchips, change the memory constraint: model weights can reside in CPU memory and be streamed on demand to MIG instances, shifting model residency from scarce HBM to abundant host memory. Leveraging this capability, we present C2CServe, a request-granularity serverless LLM serving system that allows MIG instances to switch models across requests without reloading weights into HBM. C2CServe introduces HybridGEMM, a heterogeneous-memory-aware GEMM kernel that adapts data access patterns to balance HBM and C2C bandwidth across MIG partitions using a single tuning knob. To mitigate shared-C2C contention, C2CServe further uses a hierarchical scheduler that coordinates model placement, input chunking, and kernel selection with online feedback control. On GH200, C2CServe reduces cold-start latency by up to 7.1x for dense models and 4.6x for MoE models compared with state-of-the-art serverless LLM serving systems, while maintaining over 95\% TTFT and TPOT attainment under C2C contention.

Problem

• serverless LLM serving
• cold-start latency
• MIG
• HBM
• NVLink-C2C

Innovation

• NVLink-C2C
• MIG
• serverless LLM serving
• HybridGEMM
• heterogeneous memory