AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System

2026-05-22

AI Summary

This work addresses the performance degradation in large language model (LLM) inference systems caused by divergent key-value (KV) cache lengths across requests, which introduces computational bubbles during decoding iterations, thereby reducing throughput and increasing latency. To mitigate this issue, the authors propose a prefix-aware batching strategy that groups requests with similar KV cache lengths for synchronized execution. Furthermore, they introduce a GPU-Prefetch-For-GPU architecture that integrates CPU memory buffering with batch-level scheduling to enable efficient CPU-GPU cooperative memory management. This approach significantly improves batch alignment and hardware utilization, achieving up to 1.98× higher decoding throughput and up to 7.4× lower end-to-end latency compared to state-of-the-art LLM serving systems, as demonstrated on both synthetic and real-world workloads.

Abstract

High-throughput inference serving is essential for applications built on large language models (LLMs). Existing serving frameworks reduce request-level and batch-level bubbles through batching and scheduling, but often overlook bubbles within each decode iteration. Tokens generated in the same iteration may incur different costs because they depend on KV caches of different lengths; tokens with long KV caches can become bottlenecks and delay the next iteration. We propose AlignedServe, an LLM serving framework built around prefix-aware batching. It groups requests with similar KV-cache lengths into the same batch to reduce iteration-level bubbles. To support this policy efficiently, AlignedServe uses large CPU memory to maintain sufficient in-flight requests for batching and applies a batch-level scheduling policy to reduce batch-level bubbles. It also introduces a GPU-Prefetch-For-GPU architecture, where one GPU prefetches KV cache for another to reduce CPU-to-GPU transfer latency. Experiments on synthetic and application workloads show that AlignedServe improves decoding throughput by up to 1.98 times and reduces latency by up to 7.4 times over state-of-the-art systems.

Problem

• LLM serving
• KV cache
• decode iteration bubbles
• throughput
• latency

Innovation

• prefix-aware batching
• iteration-level bubbles
• KV cache alignment
• GPU-Prefetch-For-GPU
• LLM serving