Paving the way for scientific foundation models: enhancing generalization and robustness in PDEs with constraint-aware pre-training

2025-03-24

AI Summary

Scientific foundation models (SciFMs) for partial differential equation (PDE) solving face fundamental challenges—including data scarcity, poor generalization across PDE families and physical parameters, and low robustness to observational noise. Method: We propose a novel physics-informed pretraining paradigm that leverages the PDE residual as the primary constraint signal—either as sole supervision or jointly optimized with data fidelity loss. Our framework integrates constraint-aware pretraining, residual-driven self-supervised learning, multi-task joint optimization, and neural operator architectures infused with physical priors. Contribution/Results: Experiments demonstrate consistent superiority over purely data-driven baselines across three rigorous benchmarks: transfer to unseen physical parameters, adaptation to novel PDE types, and noise-robust fine-tuning. The method achieves substantial reductions in labeled data requirements while markedly improving cross-equation generalization and solution stability.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Partial differential equations (PDEs) govern a wide range of physical systems, but solving them efficiently remains a major challenge. The idea of a scientific foundation model (SciFM) is emerging as a promising tool for learning transferable representations across diverse domains. However, SciFMs require large amounts of solution data, which may be scarce or computationally expensive to generate. To maximize generalization while reducing data dependence, we propose incorporating PDE residuals into pre-training either as the sole learning signal or in combination with data loss to compensate for limited or infeasible training data. We evaluate this constraint-aware pre-training across three key benchmarks: (i) generalization to new physics, where material properties, e.g., the diffusion coefficient, is shifted with respect to the training distribution; (ii) generalization to entirely new PDEs, requiring adaptation to different operators; and (iii) robustness against noisy fine-tuning data, ensuring stability in real-world applications. Our results show that pre-training with PDE constraints significantly enhances generalization, outperforming models trained solely on solution data across all benchmarks. These findings prove the effectiveness of our proposed constraint-aware pre-training as a crucial component for SciFMs, providing a scalable approach to data-efficient, generalizable PDE solvers.

Problem

• Enhancing generalization in PDEs with constraint-aware pre-training
• Reducing data dependence for scientific foundation models
• Improving robustness against noisy fine-tuning data

Innovation

• Incorporates PDE residuals into pre-training
• Enhances generalization with constraint-aware learning
• Reduces data dependence for SciFMs