Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces

2026-07-31

AI Summary

Existing compression methods for large language model service traces struggle to preserve critical workloads that reveal system bottlenecks and often rely on bottleneck prediction, which can induce circular evaluation issues. This work proposes the Bottleneck-Preserving Witness (BPW) framework, which identifies candidate workloads through response-agnostic feature extraction and source-side measurements, constructs a coverage-prioritized sequence, and performs prediction-free bottleneck validation via direct measurements on the target system to generate a compact yet diagnostically reliable replay test set. BPW introduces, for the first time, a multi-component bottleneck evidence preservation mechanism that operates without bottleneck prediction, leveraging a reusable hyperedge structure and dual-witness validation criteria. Experiments on BurstGPT, ServeGen, and Mooncake demonstrate that BPW achieves verification thresholds with smaller workload sets, yielding relative improvements of 2.3% in Mean Prefix Macro-F1 and 16.3% in WBRC-AUC, significantly outperforming 16 baseline strategies.

Abstract

Production LLM serving generates millions of diverse requests, making full-trace replay across serving configurations increasingly expensive. Existing trace reduction methods mainly preserve workload distributions or representative requests, but bottleneck-revealing workloads may be rare and non-representative. Moreover, evidence for one component cannot compensate for missing evidence in another, while using predicted bottlenecks as target truth creates circular evaluation. These limitations make it necessary to preserve evidence for every bottleneck component rather than rely on workload representativeness alone. We propose Bottleneck-Preserving Witnessing (BPW), a quality-constrained framework for compact and diagnostically reliable LLM serving replay suites. BPW first performs Workload Candidate Nomination using response-blind workload features and closed source-side measurements. This stage identifies workloads that may expose scheduler, prefill, decode, or KV-cache bottlenecks. Coverage-Priority Sequence Construction then organizes multi-component proposals as reusable hyperedges and prioritizes weak and uncovered dimensions. Finally, Bottleneck Truth Verification derives prediction-independent labels solely from direct target-system measurements. The verified results determine the earliest prefix satisfying the direct two-witness requirement for every component. Experiments on BurstGPT, ServeGen, and Mooncake show that BPW reaches the verified gate with a compact workload set and outperforms 16 policies, achieving relative improvements of 2.3% and 16.3% in Mean prefix Macro-F1 and WBRC-AUC, respectively. Stage-resolved and sensitivity analyses confirm the distinct contributions and local stability of its three stages. Our code is publicly available at https://github.com/llmllmllm/BPW

Problem

• LLM serving
• trace reduction
• bottleneck diagnosis
• workload representativeness
• circular evaluation

Innovation

• Bottleneck-Preserving Witnessing
• LLM serving traces
• prediction-independent verification
• workload compression
• diagnostic reliability