SpecGen: Accelerating Agentic Kernel Optimization with Speculative Generation

2026-06-16

AI Summary

This work addresses system-level bottlenecks in traditional large language model (LLM)-based automatic GPU kernel optimization, including high generation latency, sparse feedback, and low resource utilization. The authors propose a speculative generation mechanism that dynamically triggers non-inference-based candidate kernel generation during the main inference process, enabling parallel validation and profiling. This approach is augmented with dynamic GPU resource scheduling, reuse of idle video memory as remote key-value cache, and a feedback-driven early termination strategy, collectively enhancing feedback density and resource efficiency. Experiments on an H200 GPU with two inference-focused LLMs demonstrate that, under fixed time and token budgets, the proposed method substantially reduces end-to-end optimization time compared to three baselines while simultaneously increasing profiling feedback volume, resource utilization, and the final kernel speedup.

Abstract

Agentic kernel optimization automates manual GPU kernel tuning via iterative generation, validation, and profiling with reasoning LLMs, casting the optimization task as feedback-guided search. However, our workload characterization reveals three system-level inefficiencies that limit search efficiency: (1) long generation latency due to LLM reasoning, (2) insufficient profiling feedback, and (3) underutilized validation/profiling resources. Our key insight is that the ongoing reasoning generation exposes a window for producing additional candidate kernels before it completes, allowing the system to terminate reasoning early once a satisfactory kernel appears. We present SpecGen, an agentic kernel optimization system with \emph{speculative generation}. First, SpecGen forks non-reasoning generations at well-chosen trigger points in the reasoning trace to yield kernels, increasing the candidate kernel count per iteration. These kernels are validated and profiled in parallel with the ongoing reasoning, increasing profiling feedback, and keeping resources busy during generation. When a kernel meets the termination criterion, SpecGen terminates the reasoning generation early to reduce the generation latency. Second, SpecGen dynamically reallocates validation and profiling GPU pools based on the arrival rate and prioritizes requests to reduce profiling feedback latency under bursty speculative generation load. Furthermore, SpecGen utilizes spare memory of the validation/profiling GPUs as remote KV cache storage to eliminate prefix recomputation of speculative generations under limited memory budget. Experiments with two reasoning LLMs on H200 show that SpecGen reduces end-to-end time over three baseline systems, while producing more profiling feedback, increasing resource utilization, and improving kernel speedup under a fixed time and token budget.

Problem

• agentic kernel optimization
• speculative generation
• LLM reasoning
• profiling feedback
• resource utilization

Innovation

• speculative generation
• agentic kernel optimization
• reasoning LLMs
• GPU resource utilization
• early termination