SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits

2026-03-19

AI Summary

This work addresses the limitation of existing GPU kernel optimization benchmarks, which focus solely on software baseline speedups and fail to assess proximity to hardware theoretical limits. Targeting the NVIDIA Blackwell architecture, the authors construct a benchmark suite comprising 235 real-world AI model CUDA kernels and reframe the optimization objective from surpassing software baselines to approaching the hardware Speed-of-Light (SOL) ceiling. Leveraging the SOLAR framework, they derive theoretical SOL bounds for each kernel and introduce the SOL Score to quantify hardware efficiency. To ensure rigor, they design specialized kernels and a sandboxed evaluation environment supporting new data types (BF16, FP8, NVFP4), featuring clock locking, L2 cache flushing, subprocess isolation, and anti-cheating static checks. This effort establishes the first hardware-limit-oriented, cheat-resistant, and reproducible benchmark for multimodal AI kernels, providing a reliable foundation for agent-driven automated optimization.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Adversarial Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

As agentic AI systems become increasingly capable of generating and optimizing GPU kernels, progress is constrained by benchmarks that reward speedup over software baselines rather than proximity to hardware-efficient execution. We present SOL-ExecBench, a benchmark of 235 CUDA kernel optimization problems extracted from 124 production and emerging AI models spanning language, diffusion, vision, audio, video, and hybrid architectures, targeting NVIDIA Blackwell GPUs. The benchmark covers forward and backward workloads across BF16, FP8, and NVFP4, including kernels whose best performance is expected to rely on Blackwell-specific capabilities. Unlike prior benchmarks that evaluate kernels primarily relative to software implementations, SOL-ExecBench measures performance against analytically derived Speed-of-Light (SOL) bounds computed by SOLAR, our pipeline for deriving hardware-grounded SOL bounds, yielding a fixed target for hardware-efficient optimization. We report a SOL Score that quantifies how much of the gap between a release-defined scoring baseline and the hardware SOL bound a candidate kernel closes. To support robust evaluation of agentic optimizers, we additionally provide a sandboxed harness with GPU clock locking, L2 cache clearing, isolated subprocess execution, and static analysis based checks against common reward-hacking strategies. SOL-ExecBench reframes GPU kernel benchmarking from beating a mutable software baseline to closing the remaining gap to hardware Speed-of-Light.

Problem

• GPU kernel optimization
• hardware efficiency
• benchmarking
• Speed-of-Light
• agentic AI

Innovation

• Speed-of-Light benchmarking
• GPU kernel optimization
• hardware efficiency
• SOL Score
• agentic AI evaluation