LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

2026-08-08

AI Summary

This work addresses the challenge of accurately attributing latency costs to individual tenant requests in multi-tenant large language model serving, where request batching improves throughput but obscures per-request latency accounting. The authors propose a roofline-model-based method for real-time latency attribution that, for the first time, enables lightweight and high-fidelity decomposition of per-request latency within the scheduling loop. Their approach models batched inference latency as a piecewise-linear function proportional to FLOPs and memory I/O. Experiments on Llama 3.1-8B and Qwen 2.5-14B/32B demonstrate substantial improvements over token-count baselines: in the prefill phase, p90/p99 relative errors are reduced by up to 2.5×/3.3×, and in the decode phase by up to 3.5×/4.4×, with R² values approaching 1. These results significantly advance fine-grained resource metering and control in shared LLM inference systems.

Abstract

As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control. Enabling fractional sharing of the inference engine requires a real-time, per-request attribution primitive that is accurate and light enough to run inside the scheduling loop. We present LLMVisor, a roofline-guided latency attribution model that captures the memory-bound and compute-bound phases via a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic. LLMVisor decomposes batch latency into additive, per-request shares and runs efficiently at microsecond scale. We evaluate LLMVisor across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs under varying tensor parallelism and workload mixes. Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x and 3.3x at p90 and p99, respectively, for prefill, and by up to 3.5x and 4.4x for decode, despite batching variability and sequence divergence.

Problem

• multi-tenant LLM serving
• latency attribution
• co-batching
• resource accounting
• inference latency

Innovation

• latency attribution
• multi-tenant LLM serving
• roofline model
• real-time scheduling
• fractional GPU sharing