Introduction to optimization methods for training SciML models

2026-01-15

AI Summary

This work addresses the challenges posed by loss functions in scientific machine learning (SciML), which are often globally coupled, stiff, and strongly anisotropic, thereby undermining conventional optimization methods. The paper systematically reviews first- and second-order optimization techniques tailored for SciML—including adaptive gradient methods, curvature-aware approaches, and deterministic strategies—and highlights the dominant influence of the spectral properties of underlying physical models on optimization dynamics. By unifying the optimization frameworks of classical machine learning and SciML, the study integrates physical constraints with data-driven modeling to offer reproducible practical guidelines for efficient optimization. Furthermore, it delineates key open problems at this interdisciplinary frontier, providing a roadmap for future research.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Web Mining and Content Analysis: Machine learning and data science for the Web
• Security and Privacy: Security and privacy of machine learning and AI applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Optimization is central to both modern machine learning (ML) and scientific machine learning (SciML), yet the structure of the underlying optimization problems differs substantially across these domains. Classical ML typically relies on stochastic, sample-separable objectives that favor first-order and adaptive gradient methods. In contrast, SciML often involves physics-informed or operator-constrained formulations in which differential operators induce global coupling, stiffness, and strong anisotropy in the loss landscape. As a result, optimization behavior in SciML is governed by the spectral properties of the underlying physical models rather than by data statistics, frequently limiting the effectiveness of standard stochastic methods and motivating deterministic or curvature-aware approaches. This document provides a unified introduction to optimization methods in ML and SciML, emphasizing how problem structure shapes algorithmic choices. We review first- and second-order optimization techniques in both deterministic and stochastic settings, discuss their adaptation to physics-constrained and data-driven SciML models, and illustrate practical strategies through tutorial examples, while highlighting open research directions at the interface of scientific computing and scientific machine learning.

Problem

• scientific machine learning
• optimization
• physics-informed
• differential operators
• loss landscape

Innovation

• scientific machine learning
• physics-informed optimization
• curvature-aware methods
• loss landscape anisotropy
• deterministic optimization