PerfReasoning: How Well Do LLMs Reason on Hardware Performance?

2026-09-03

AI Summary

本文通过引入PerfReasoning基准，评估大型语言模型在硬件性能推理及生成性能模型代码方面的能力，揭示了模型在性能建模上的局限性。

Abstract

Performance modeling is central to hardware design and software optimization, yet constructing these models requires structured reasoning about computation, data reuse, storage, and movement. We introduce PerfReasoning, a benchmark that evaluates LLMs both as direct performance reasoners and as generators of analytical performance-model code. Given workload, architecture, and mapping specifications, models compare mappings and predict off-chip traffic and buffer requirements. The strongest closed-source models exceed 90% on reasoning-based Q&A, and the best open-weight model reaches 82.4%. However, model construction is substantially harder: while GPT-5.6 Sol exceeds 80% pass rate, all other model configurations average below 15% and vary markedly across runs. Task-specific RL raises a 4B model's mapping-reasoning accuracy by 15.7 points, whereas feedback-free multi-round self-revision prompting is not reliably effective. PerfReasoning exposes the gap between plausible architectural reasoning and reliable performance-model construction. We will publicly release the benchmark to support reproducible evaluation and track future progress.

Problem

• Performance Modeling
• LLMs
• Hardware Performance
• Analytical Performance-Model Code
• Workload and Architecture Specifications

Innovation

• PerfReasoning
• performance reasoning
• analytical performance-model code
• reinforcement learning
• model construction