numerical experiment design and validation

Scientific software selection frequently suffers from non-reproducible benchmarks due to multi-library, multi-metric evaluation and dynamic evolution—such as the introduction of new algorithms or modifications to test cases and evaluation criteria. This paper addresses numerical integration over arbitrary 2D/3D domains with implicit or parameterized boundaries (cut-cell quadrature), proposing the first automated benchmarking framework that systematically integrates CI/CD engineering practices into scientific computing workflows. The framework unifies GitHub Actions, Docker, Python-based scheduling, Jupyter-based report generation, and semantically versioned result archiving. It supports automated configuration, execution, visualization, and historical result comparison. It achieves >90% automation for benchmark tasks and regression detection; reduces integration time for new libraries or algorithms by 70%; and enables precise attribution of performance deviations to specific code commits. The framework significantly enhances reliability, reproducibility, and evolutionary adaptability in scientific software evaluation.

This study addresses the widespread lack of computational reproducibility in R supplementary code deposited on the Open Science Framework (OSF). A systematic audit of 296 published R code packages revealed that 98.8% incompletely declare dependencies. To address this, we propose the first automated reproducibility auditing framework tailored to the R ecosystem. It combines static source-code analysis—leveraging regular expressions and abstract syntax trees (ASTs)—to accurately infer dependencies, with Docker-based containerized execution and failure diagnostics (e.g., path errors, OS-specific inconsistencies, missing packages) to enable end-to-end environment reconstruction and validation. Experiments successfully executed 25.87% of scripts, identifying undeclared dependencies, hardcoded file paths, and cross-platform compatibility issues as the three primary barriers to reproducibility. The framework enables large-scale, low-cost, and scalable quantitative assessment of computational reproducibility in scholarly research, providing a practical toolchain to enhance transparency and verifiability.

Scientific computing artifacts—such as analysis scripts and software prototypes—frequently suffer from poor reproducibility due to environmental heterogeneity, dependency drift, and inadequate documentation, thereby undermining research credibility. To address this, we introduce the first cross-disciplinary, structured, and standardized benchmark dataset for computational experiments, encompassing workflows ranging from single-script executions to multi-language, complex pipelines. Our framework uniformly models metadata, standardizes dependency declarations (e.g., requirements.txt, Dockerfiles), encapsulates multi-language execution procedures, and prescribes a rigorous documentation protocol. The dataset comprises dozens of human-validated, fully reproducible experimental cases, enabling objective, comparable, and reproducible evaluation of reproducibility tools. This work fills a critical gap in the field by providing the first systematic, community-grounded benchmark for assessing computational reproducibility, thereby significantly enhancing the rigor, transparency, and comparability of reproducibility research.

This paper addresses core challenges to computational experiment reproducibility in HPC and systems research—namely, strong hardware dependence, complex environment configuration, and high operational costs. Methodologically, it introduces a layered, pragmatic solution framework grounded in a novel “feasibility–rigor trade-off” principle, and proposes a dual-dimensional (challenge–recommendation) model tailored for authors, reviewers, institutions, and the broader community. The approach integrates structured workshops, cross-role consensus modeling, actionable checklists, and explores ecosystem innovations including AI-assisted environment generation and artifact digital repositories. Key contributions include: (1) a comprehensive, lifecycle-spanning practical guide and standardized reproducibility checklist; (2) a widely adopted community consensus on reproducible practices within the HPC domain; and (3) a systemic pathway advancing reproducibility from conceptual principle to institutionalized practice.

This work addresses the high prevalence of silent failures in AI-generated scientific simulation code when applied to non-textbook problems, which undermines its reliability. To bridge this trust gap, we propose the Judge Agent framework, which systematically validates well-posedness, convergence, and error bounds through automated verification. We introduce a simulatability class \( \mathcal{S} \) and a structured, solver-agnostic specification format (spec.md) that enables machine-readable problem descriptions. Evaluated across 134 cases spanning 12 scientific domains, our approach reduces the silent failure rate from 42% to 1.5%. In 72 blind test tasks, it achieves an 89% success rate, and in clinical CT reconstruction experiments, it reaches 99% of expert-level performance, substantially enhancing the credibility of AI-generated scientific code.

This work addresses the absence of a standardized benchmark for evaluating code generation targeting partial differential equation (PDE) solvers, particularly with respect to numerical accuracy, computational efficiency, and compatibility with mainstream finite element libraries. To bridge this gap, the authors introduce the first multi-metric, multi-library benchmark for PDE solver generation, comprising 645 structured instances spanning six mathematical problem types and eleven PDE classes. The benchmark supports three major finite element frameworks—DOLFINx, Firedrake, and deal.II—and incorporates a staged evaluation framework that holistically assesses code executability, numerical correctness, and performance. Experimental results demonstrate that while current large language models can produce executable code, their success rate drops substantially when stringent accuracy and efficiency requirements are imposed, thereby underscoring the necessity and effectiveness of the proposed benchmark in advancing reliable and efficient automated PDE solver generation.

This work addresses the frequent mismatch between user-specified physical intent and the actual behavior of multiphysics simulation code generated by large language models, often due to erroneous implementations of partial differential equations (PDEs). To bridge this gap, we propose a PDE-structure-based intent verification method that deterministically reconstructs the governing equations implicitly encoded in the generated code and compares them against the user’s intended PDEs, enabling semantic correctness validation and iterative refinement. We introduce, for the first time, a formal metric termed the Intent Fidelity Score (IFS) to quantify alignment with physical intent, establish a PDE-driven feedback loop, and demonstrate compatibility with major PDE frameworks including MOOSE, FEniCS, and FreeFEM. Evaluated on 220 cases in MooseBench, our approach substantially improves IFS—by 0.22–0.41 on challenging instances with initial IFS < 0.7—while audits reveal that execution-only repair strategies still yield physically incorrect results in 39–40% of cases.

Ensuring correctness and reliability of large-scale scientific software remains highly challenging due to the complexity of numerical algorithms, floating-point uncertainties, and domain-specific modeling requirements. Method: This project introduces a novel “multi-tool collaborative verification” paradigm, driven by challenging scientific computing problems and integrating deductive verification, floating-point error analysis, coupled model specification, and domain-aware testing—orchestrated through a peer-review–driven expert collaboration mechanism. Contribution/Results: The work yields five peer-reviewed publications, three invited talks, and a curated benchmark suite of challenge problems. It systematically identifies current technical bottlenecks and methodological boundaries, and—uniquely—establishes a comprehensive, end-to-end trustworthiness roadmap spanning modeling, implementation, and verification stages. This roadmap provides both theoretical foundations and practical frameworks for cross-disciplinary integration and standardization of scientific software verification tools.

Existing benchmarks inadequately assess LLM agents’ scientific development capabilities within real-world research software ecosystems: neither conceptual reasoning nor general programming benchmarks cover end-to-end collaborative evolution of production-grade scientific code. To address this, we propose AInsteinBench—the first LLM agent evaluation benchmark grounded in authentic research software ecosystems. It targets six domains—including quantum chemistry and molecular dynamics—using maintenance-level Pull Requests from widely adopted open-source libraries. Tasks execute in reproducible sandbox environments and integrate test-driven validation, scientific-semantic failure analysis, unit test coverage measurement, and difficulty calibration. Innovatively anchoring evaluation in production codebases, AInsteinBench introduces expert review and multi-stage filtering to define and quantify core competencies of scientific computing agents: domain-knowledge integration, numerical robustness, and collaborative code evolution. Our systematic evaluation exposes critical weaknesses of leading code-generation agents across these dimensions.

This study investigates the use of large language models (LLMs) to automatically translate neutral graph representations of fluid systems into high-quality, functionally correct code executable in mainstream simulation environments such as WNTR and Modelica. The authors systematically evaluate ten state-of-the-art LLMs combined with six prompting strategies across multiple benchmark scenarios, assessing generated code through software quality metrics and simulation fidelity. This work presents the first systematic comparison in the domain of fluid system modeling that examines how different LLMs and prompt engineering techniques influence both syntactic correctness and functional fidelity of generated simulation code, offering empirical guidance for model-driven code generation. Experimental results demonstrate that optimal configurations can produce syntactically valid code; however, a significant gap remains in achieving high simulation fidelity, highlighting key directions for future improvement.