Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving

2026-06-28

AI Summary

This work addresses the lack of systematic understanding in co-designing the prefill and decode phases for heterogeneous large language model inference, which hinders deployment efficiency. Focusing on four key design axes—accelerator architecture, precision, interconnect, and KV residency—the study reveals that only a subset of these factors forms strongly coupled constraints and identifies three pivotal boundary decisions: compute placement, KV representation, and KV ownership. Integrating empirical insights from industrial deployments with runtime analysis, the authors propose a runtime role–driven precision strategy, a byte-level KV transfer mechanism, and explicit ownership management to construct a multi-axis design space model. The resulting framework yields actionable design guidelines that substantially improve heterogeneous inference efficiency.

Abstract

Heterogeneous prefill-decode (PD) inference is now in production: prefill on cost-efficient or supply-available accelerators, decode on bandwidth-strong ones, and KV state crossing mixed interconnects in mixed numerical formats. Each deployment makes these decisions on its own. What is missing is the picture across configurations-which decisions must be made jointly at the PD boundary, and which can be made independently. We propose a design space organized along four design axes-accelerator, precision, interconnect, and KV residency and the workload regime (stage pressure) they respond to. We show that only a subset of interactions among these factors become binding constraints once PD inference becomes heterogeneous. These interactions surface through three recurring boundary decisions: compute placement, KV representation, and KV ownership. The resulting analysis yields concrete guidance. Precision policy belongs to runtime roles rather than to a single system-wide setting, because the same low-bit format relieves different bottlenecks on each side of the boundary. KV transfer engines move bytes rather than tensor semantics, making representation compatibility an explicit boundary concern whenever producer and consumer differ. The KV handoff also carries a lifecycle-reservation, release, and failure recovery-that spans prefill and decode and requires explicit ownership. Two further interactions remain open. Cross-vendor and interconnect-related claims are stated as design guidance grounded in industrial deployment observations and source-code inspection of the runtimes involved.

Problem

• heterogeneous LLM inference
• prefill-decode boundary
• design space
• KV state management
• system deployment

Innovation

• heterogeneous LLM inference
• prefill-decode boundary
• KV cache management
• precision policy
• design space analysis