Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

2026-09-16

AI Summary

本文提出FairInference，通过设定每令牌的截止时间来解决多租户LLM服务中的延迟隔离问题，保证了良好行为客户端的令牌级延迟，并提高了整体吞吐量。

Abstract

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel δ-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + δ time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.

Problem

• Latency SLO
• Performance Isolation
• Multi-Tenant LLM Serving
• Token-Level Latencies

Innovation

• Token Latency Fairness
• Multi-Tenant LLM Serving
• δ-token Guarantee
• Resource Sharing Delay Bounding