Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

2026-04-14

AI Summary

Existing benchmarks for large language model agents are predominantly confined to binary success-or-failure tasks, failing to capture the iterative refinement of feasible solutions central to real-world engineering. To address this limitation, this work introduces the first generative optimization benchmark tailored to open-ended, complex engineering problems, encompassing 47 industrial-scale simulation tasks across five domains. The benchmark incorporates continuous rewards and hard feasibility constraints, and implements a “propose–execute–evaluate” iterative framework to assess agents’ capacity for sustained improvement under a fixed interaction budget. Experimental results show that Claude 4.6 Opus achieves the best performance, yet the tasks remain challenging overall. Both the frequency and magnitude of improvements exhibit power-law decay, with deep iteration proving particularly critical for performance gains.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Prompt Engineering / Prompting
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning $47$ tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while Claude 4.6 Opus achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency ($\sim$ 1/iteration) and magnitude ($\sim$ 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems.

Problem

• generative optimization
• engineering tasks
• LLM agents
• benchmarking
• iterative design

Innovation

• generative optimization
• self-evolving agents
• engineering benchmark
• executable feedback
• iterative design