RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

2026-08-12

AI Summary

This work addresses three key limitations of existing benchmarks for evaluating large language models (LLMs) in generating Triton kernels: their restriction to PyTorch-to-Triton translation, isolated assessment of individual kernel performance, and reliance on error-prone manual evaluation. To overcome these shortcomings, the authors propose the first benchmark constructed from real-world pull requests in open-source projects, extracting authentic modification tasks from mainstream AI frameworks. By integrating natural language requirement parsing with automated end-to-end integration testing, this benchmark enables production-grade, reproducible evaluation of LLM-generated Triton kernels. Experimental results demonstrate that current state-of-the-art LLMs still exhibit significant deficiencies in generating correct and efficient Triton kernels under realistic development scenarios.

Abstract

In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.

Problem

• Triton kernel generation
• LLM benchmarking
• real-world AI frameworks
• end-to-end evaluation
• GPU kernel

Innovation

• Triton kernel generation
• real-world benchmark
• end-to-end evaluation
• LLM for GPU kernels
• AI framework integration