KernelSight-LM: A Kernel-Level LLM Inference Simulator

2026-06-26

AI Summary

This work addresses the need for efficient performance evaluation of large language model inference across diverse hardware and scheduling configurations, a task hindered by the high cost and poor generalizability of end-to-end empirical measurements. The authors propose a fine-grained inference simulator that decomposes computation kernels, communication, and host overheads using a roofline model and employs a discrete-event scheduler to accurately model optimizations such as prefix caching and continuous batching, enabling token-level execution tracing and kernel-level latency breakdown. Innovatively, a two-tier prediction mechanism is introduced, which achieves high-fidelity latency prediction across GPU generations without requiring target-device measurements or with only a single model-agnostic microbenchmark. Experiments demonstrate median errors of just 15.4% for time-to-first-token (TTFT), 12.8% for time-per-output-token (TPOT), and 3.0% for throughput—approaching the accuracy of specialized profiling tools while substantially reducing measurement overhead.

Abstract

As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets. However, the end-to-end behavior of LLMs couples serving-layer policies with low-level GPU kernel execution and rapidly evolving architectures, forcing slow, deployment-specific benchmarking that is hard to generalize. We present KernelSight-LM, a fine-grained inference simulator that models token-level execution and produces kernel-level latency breakdowns. It decomposes each serving step into a roofline kernel model with a learned efficiency term, a communication model, and a host-overhead model, composed through a discrete-event scheduler that also captures mechanisms like prefix caching and continuous batching. KernelSight-LM offers two prediction tiers that trade target-GPU data for accuracy. The cross-generation tier uses no target-GPU measurements, only hardware specifications and kernel microbenchmarks from previously profiled GPUs, and predicts per-kernel latency on an unseen GPU generation to 12.1% error, a 1.8x improvement over the roofline baseline (22.0%). A second target-measured tier adds one model-agnostic kernel-microbenchmark sweep on the target GPU, sharpening per-kernel error to 3.8%, a 7.3x improvement over a comparable baseline (27.7%). Both tiers require far less target-GPU data than the prior systems they extend. In our simulator, these predictions yield end-to-end median (p50) errors across six model families of 15.4%, 12.8%, and 3.0% (TTFT, TPOT, throughput) in the cross-generation tier and 14.3%, 6.2%, and 2.7% in the target-measured tier, matching dedicated profiling tools while collecting far less on-device data. Beyond prediction, its kernel-level bottleneck breakdowns support hardware/software co-design and capacity planning.

Problem

• LLM inference
• performance prediction
• hardware generalization
• latency estimation
• benchmarking

Innovation

• LLM inference simulation
• kernel-level modeling
• roofline model
• cross-generation prediction
• discrete-event scheduling