SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving

2025-11-29

AI Summary

Large language models (LLMs) deployed with tensor parallelism (TP) and pipeline parallelism (PP) face a critical bottleneck in sampling (logits-to-token decoding), which limits end-to-end throughput and latency: sampling is inherently non-scalable, load-unbalanced, and its relative overhead grows as GPU compute accelerates. This paper introduces the first **decoupled sequence-parallel sampling architecture**, fully offloading sampling to the CPU to establish an independent, stage-agnostic, zero-modification decision plane. Key innovations include column-wise CPU-based penalty application and truncation-aware vocabulary filtering, speculative hot-vocabulary sampling with rejection correction (SHVS), and dynamic hot-vocabulary sizing modeling. The design seamlessly integrates with existing data-plane optimizations (e.g., KV caching, attention/GEMM acceleration). Experiments demonstrate up to 96% higher end-to-end throughput, 20–65% lower P95 latency, and near-zero sampling overhead—significantly breaking the performance ceiling of distributed LLM serving.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) scale out with tensor parallelism (TP) and pipeline parallelism (PP) and production stacks have aggressively optimized the data plane (attention/GEMM and KV cache), sampling, the decision plane that turns logits into tokens, becomes a new bottleneck. This creates a structural holdout: sampling neither expands with TP nor balances across PP stages, so its share of iteration time grows as GPUs get faster and it caps pipeline frequency at the last stage. We present SIMPLE, a stage-agnostic, sequence-parallel, overlappable decision plane that disaggregates sampling into a CPU-side service and shrinks its runtime footprint back to a minor, hidden role. SIMPLE combines: (1) sequence-parallel sampling, which shards work along the batch dimension and removes vocabulary-axis collectives; (2) a CPU-based algorithm with column-wise penalties and truncation-first filtering to realize single-pass, linear-time kernels; and (3) speculative hot-vocab sampling (SHVS), which samples on a small hot set with rejection-correctness and uses a simple sizing model to choose the hot-vocab size that maximizes throughput. In evaluation, SIMPLE improves end-to-end throughput by up to 96% and reduces P95 latency by 20-65%. Crucially, SIMPLE requires no user-side code changes and composes with existing data-plane optimizations, unlocking scaling benefits that compound with future GPU generations.

Problem

• Distributed LLM serving bottleneck in sampling decision plane
• Sampling cannot scale with tensor or pipeline parallelism
• Sampling limits pipeline frequency at final stage

Innovation

• Disaggregates sampling to CPU service to hide runtime overhead
• Uses sequence-parallel sampling removing vocabulary collectives
• Implements speculative hot-vocab sampling with rejection correction