Towards Richer Challenge Problems for Scientific Computing Correctness

2025-10-15

AI Summary

Formal methods (FM) and programming language (PL) techniques struggle to verify correctness in scientific computing (SC) due to a lack of consensus between FM/PL communities and SC practitioners on what constitutes verifiable correctness—particularly regarding numerical precision, algorithmic stability, and other SC-specific dimensions. Method: We propose the first systematic challenge-problem framework tailored to SC: (1) we distill core correctness dimensions intrinsic to SC; (2) we establish principled design criteria and evaluation metrics for challenge problems; and (3) we construct a benchmark suite comprising formally verifiable, reproducible problems grounded in real-world FM/PL techniques. Contribution/Results: The framework bridges the semantic gap between SC and FM/PL, providing a targeted, rigorous testbed for verification tools. It enables more realistic, application-grounded assessment of verification techniques’ applicability and effectiveness in production scientific software, thereby advancing trustworthy scientific computation.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

Correctness in scientific computing (SC) is gaining increasing attention in the formal methods (FM) and programming languages (PL) community. Existing PL/FM verification techniques struggle with the complexities of realistic SC applications. Part of the problem is a lack of a common understanding between the SC and PL/FM communities of machine-verifiable correctness challenges and dimensions of correctness in SC applications. To address this gap, we call for specialized challenge problems to inform the development and evaluation of FM/PL verification techniques for correctness in SC. These specialized challenges are intended to augment existing problems studied by FM/PL researchers for general programs to ensure the needs of SC applications can be met. We propose several dimensions of correctness relevant to scientific computing, and discuss some guidelines and criteria for designing challenge problems to evaluate correctness in scientific computing.

Problem

• Addressing the gap between scientific computing and verification communities
• Developing specialized challenge problems for scientific computing correctness
• Establishing correctness dimensions and guidelines for verification techniques

Innovation

• Propose specialized challenge problems for verification
• Define correctness dimensions for scientific computing
• Establish guidelines for evaluating scientific computing correctness