CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

2026-07-08

AI Summary

Existing software fails to effectively exploit the shared-memory architecture of multi-GPU systems, and conventional tensor parallelism struggles to meet the low-latency inference demands of large language models. This work proposes CTA-pipelining, a novel approach that introduces CTA-level pipelining as an orthogonal dimension independent of tensor parallelism. By explicitly modeling CTA-level dependencies, the method enables concurrent execution of GPU kernels across devices in shared-memory multi-GPU systems. Built upon CUTLASS, cuBLAS, and NCCL, our fine-grained pipelined scheduling is implemented on H200 and B200 platforms. Evaluated on a two-layer GEMM representative of MLP operations, the approach reduces inference latency by 31.8% and 29.6% compared to micro-batching and tensor parallelism, respectively.

Abstract

The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from being throughput-driven to latency-bound, necessitating latency-oriented scaling methods beyond Tensor Parallelism (TP). Thus, we introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems. As a latency-oriented spatial scaling technique, CTA-pipelining leverages dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. We demonstrate its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems. Results show on 2-layer GEMM, representing the MLP operation, CTA-pipelining reduces latency by up to 31.8% compared to micro-batching, and 29.6% compared to TP. It can also be combined with TP as an orthogonal scaling dimension to further push the latency boundary.

Problem

• latency-oriented
• multi-GPU systems
• spatial scaling
• Large Language Models
• Tensor Parallelism

Innovation

• CTA-pipelining
• latency-oriented scaling
• multi-GPU shared-memory
• spatial scaling
• Cooperative Thread Array