CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs

2026-05-08

AI Summary

This work systematically evaluates the capability of large language models to generate expert-level, hardware-aware CUDA code, revealing a substantial gap between current model performance and human expert optimization. We introduce a multi-tiered benchmark encompassing kernel functions, operators, full applications, and open-ended challenge tasks, evaluated across Ampere, Hopper, and Blackwell GPU architectures. For the first time, end-to-end expert-level CUDA optimization is established as a formal evaluation criterion, combining GPU-specific microbenchmarks with domain-aware semantic validators. Experimental results demonstrate that while existing models—such as Claude-Opus-4.6 and GPT-5.4—can produce runnable code, they rarely reproduce critical optimization strategies employed by experts. Performance further degrades when application-level semantics are incorporated, underscoring fundamental limitations in current models’ hardware reasoning and tool integration capabilities.

Abstract

Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware optimization. We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems. It spans single kernels, module-level operators, full applications, and unsolved challenge tasks across Ampere, Hopper, and Blackwell GPUs, with end-to-end tasks gated by domain-specific semantic validators. Evaluating models such as Claude-Opus-4.6 and GPT-5.4 shows a large gap between runnable CUDA and expert CUDA engineering: models often compile and pass tests, but rarely recover the optimization strategies needed to match expert performance. Application semantics further reduce success, and iterative or tool-augmented feedback can improve correctness while drifting toward slow fallback implementations. These results show that automated CUDA programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives that connect code understanding to hardware architecture-grounded intelligence.

Problem

• CUDA optimization
• large language models
• hardware-aware programming
• GPU architectures
• expert-level code generation

Innovation

• CUDA optimization
• hardware-aware benchmarking
• large language models
• expert-level code generation
• GPU architecture