A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning

2025-09-17

AI Summary

Residual-adaptive strategies in scientific machine learning have long relied on heuristic designs and lack a unified theoretical foundation. Method: This paper proposes the first variational-principles-based residual-adaptive framework. By introducing convex transformations to structurally model residuals, it unifies weighting mechanisms and sampling distribution optimization within the variational minimization of an error functional, thereby establishing a theoretical link between adaptive weights and error norms. The method supports diverse weighting schemes (e.g., exponential, linear) and stochastic optimization algorithms, significantly reducing loss estimation variance and improving gradient signal-to-noise ratio. Contribution/Results: Experiments demonstrate consistent improvements in convergence speed and accuracy for neural PDE solvers and operator learning tasks. The framework exhibits strong generalization across optimizers and network architectures, providing an interpretable, analytically tractable theoretical foundation for residual-driven scientific AI.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Residual-based adaptive strategies are widely used in scientific machine learning but remain largely heuristic. We introduce a unifying variational framework that formalizes these methods by integrating convex transformations of the residual. Different transformations correspond to distinct objective functionals: exponential weights target the minimization of uniform error, while linear weights recover the minimization of quadratic error. Within this perspective, adaptive weighting is equivalent to selecting sampling distributions that optimize the primal objective, thereby linking discretization choices directly to error metrics. This principled approach yields three benefits: (1) it enables systematic design of adaptive schemes across norms, (2) reduces discretization error through variance reduction of the loss estimator, and (3) enhances learning dynamics by improving the gradient signal-to-noise ratio. Extending the framework to operator learning, we demonstrate substantial performance gains across optimizers and architectures. Our results provide a theoretical justification of residual-based adaptivity and establish a foundation for principled discretization and training strategies.

Problem

• Formalizing residual-based adaptive strategies in neural PDE solvers
• Linking discretization choices directly to error metrics
• Enhancing learning dynamics and reducing discretization error

Innovation

• Variational framework integrates residual transformations
• Adaptive weighting optimizes primal objective via sampling
• Reduces discretization error through variance reduction