PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers

2026-05-03

AI Summary

This work addresses the challenge of maximizing throughput for offline large language model inference on commercial GPU servers under fixed resource budgets. The authors propose a co-design approach that, for the first time, integrates pipeline parallelism with KV cache offloading. By leveraging the low communication overhead and time-multiplexed activation characteristics of pipelining, the method dynamically expands effective GPU memory capacity and enables coordinated scheduling between computation and data transfer. Evaluated on an 8-GPU node, the proposed system achieves 2.51× the throughput of vLLM and outperforms other state-of-the-art systems by factors of 1.42× and 1.38×, respectively, substantially improving offline inference efficiency under constrained interconnect bandwidth and GPU memory conditions.

Abstract

Offline LLM inference seeks to maximize request processing under fixed budgets, making commodity GPU servers a promising choice. However, prior work typically considers offloading and parallelism in isolation, resulting in suboptimal performance. In this paper, we propose PipeMax, a high-throughput LLM inference system that integrates pipeline parallelism with offloading to overcome interconnect and memory constraints on GPU servers. Particularly, pipeline parallelism naturally incurs low communication overhead and keeps only one batch active on each GPU at a time, which enables offloading the KV cache of inactive batches. By coordinating computation with offloading data movement, PipeMax effectively expands GPU memory capacity and sustains large-batch execution. Experiments show that PipeMax achieves up to 2.51x higher throughput than vLLM, and up to 1.42x and 1.38x higher throughput than state-of-the-art high-throughput LLM systems, respectively, on an 8-GPU node.

Problem

• offline LLM inference
• commodity GPU servers
• throughput
• memory constraints
• interconnect limitations

Innovation

• pipeline parallelism
• KV cache offloading
• LLM inference
• commodity GPU servers
• high-throughput