PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation

2026-05-10

AI Summary

This work addresses the absence of a standardized benchmark for evaluating code generation targeting partial differential equation (PDE) solvers, particularly with respect to numerical accuracy, computational efficiency, and compatibility with mainstream finite element libraries. To bridge this gap, the authors introduce the first multi-metric, multi-library benchmark for PDE solver generation, comprising 645 structured instances spanning six mathematical problem types and eleven PDE classes. The benchmark supports three major finite element frameworks—DOLFINx, Firedrake, and deal.II—and incorporates a staged evaluation framework that holistically assesses code executability, numerical correctness, and performance. Experimental results demonstrate that while current large language models can produce executable code, their success rate drops substantially when stringent accuracy and efficiency requirements are imposed, thereby underscoring the necessity and effectiveness of the proposed benchmark in advancing reliable and efficient automated PDE solver generation.

Abstract

PDE-to-solver code generation aims to automatically synthesize executable numerical solvers from partial differential equation (PDE) specifications. This task requires not only understanding the mathematical structure of PDEs, but also selecting appropriate discretization schemes and solver configurations, and correctly implementing the resulting formulations in finite-element method (FEM) libraries. Existing code generation benchmarks mainly evaluate syntactic correctness, or success on predefined test cases. To our knowledge, there is currently no publicly available benchmark specifically for PDE-to-solver code generation, and general-purpose code benchmarks do not fully capture the unique challenges of numerical PDE solution, such as ensuring solver accuracy, efficiency, and compatibility with professional FEM libraries. We introduce PDEAgent-Bench, to the best of our knowledge, the first multi-metric, multi-library benchmark for PDE-to-solver code generation. PDEAgent-Bench contains 645 instances across 6 mathematical categories and 11 PDE families, with common FEM libraries for DOLFINx, Firedrake, and deal.II. Each instance provides an agent-facing problem specification, a reference solution on a prescribed evaluation grid, and case-specific accuracy and runtime targets. PDEAgent-Bench adopts a staged evaluation framework in which generated solvers must sequentially pass executability, numerical accuracy, and computational efficiency checks. Experiments with representative LLMs and code agents show that models can often produce runnable code, but their pass rate drops substantially once accuracy and efficiency requirements are enforced. These results indicate that current agents remain limited in producing numerically reliable and efficient PDE solvers, and that PDEAgent-Bench provides a reproducible testbed grounded in the practical requirements of numerical PDE solving.

Problem

• PDE-to-solver code generation
• numerical PDE solving
• finite-element method
• code generation benchmark
• solver accuracy

Innovation

• PDE-to-solver code generation
• multi-metric benchmark
• finite element method (FEM)
• numerical accuracy
• computational efficiency