$Φ$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

2026-09-09

AI Summary

本文通过提出Φ-Bench解决大型语言模型在基础设施工程中长期、开放式任务评估不足的问题，该基准测试覆盖了从局部内核到系统优化的不同复杂度任务。

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present $Φ$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories, $Φ$-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.

Problem

• Large Language Models
• Infrastructure Engineering
• Benchmarking
• Open-ended Tasks
• Long-horizon Optimization

Innovation

• Large Language Models
• Infrastructure Engineering
• Benchmarking
• End-to-End Optimization
• Complex Task Completion