Verification and Validation for Trustworthy Scientific Machine Learning

2025-02-21

AI Summary

The rapid proliferation of scientific machine learning (SciML) applications is hindered by limited trustworthiness due to the absence of systematic modeling standards, particularly for verification and validation (V&V) of predictive SciML models in physical system modeling. Method: This work systematically adapts the established V&V paradigm from traditional computational science to the SciML context, proposing sixteen interdisciplinary, consensus-driven best practices. The approach integrates solution accuracy verification, experimental calibration, uncertainty quantification, interpretability analysis, and rigorous domain-knowledge embedding. Contribution/Results: It establishes the first comprehensive, practice-oriented framework for trustworthy SciML. The framework has been widely adopted across disciplines and has catalyzed standardization initiatives at major institutions—including NASA and the U.S. Department of Energy (DOE). Furthermore, it has enabled three high-fidelity simulation models to successfully pass formal V&V review, demonstrating its practical efficacy and scalability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Scientific machine learning (SciML) models are transforming many scientific disciplines. However, the development of good modeling practices to increase the trustworthiness of SciML has lagged behind its application, limiting its potential impact. The goal of this paper is to start a discussion on establishing consensus-based good practices for predictive SciML. We identify key challenges in applying existing computational science and engineering guidelines, such as verification and validation protocols, and provide recommendations to address these challenges. Our discussion focuses on predictive SciML, which uses machine learning models to learn, improve, and accelerate numerical simulations of physical systems. While centered on predictive applications, our 16 recommendations aim to help researchers conduc

Problem

• Establish trustworthy SciML practices
• Address verification and validation challenges
• Improve predictive SciML model reliability

Innovation

• Establish consensus-based good practices
• Apply verification and validation protocols
• Enhance predictive SciML model trustworthiness