Discovering Physical Directions in Weight Space: Composing Neural PDE Experts

2026-05-14

AI Summary

This work investigates whether weight updates in neural PDE operators after multi-physics fine-tuning encode reusable physical structures—a question that remains unclear. Starting from a shared base model, the authors fine-tune on both low- and high-fidelity physical scenarios and, for the first time, decompose the resulting weight updates into shared adaptation and physics-aligned components. Building on this decomposition, they propose a training-free expert fusion mechanism (CCM) that integrates low-rank fine-tuning, physics-aware coordinate mapping, and posterior fusion to enable cross-PDE transfer across architectures such as FNO and DPOT. Experiments demonstrate significant improvements in out-of-distribution generalization: rollback errors in extrapolation scenarios are reduced by 54.2%, 42.8%, and 13.8% on reaction–diffusion, Navier–Stokes, and dam-break problems, respectively.

Abstract

Recent advances in neural operators have made partial differential equation (PDE) surrogate modeling increasingly scalable and transferable through large-scale pretraining and in-context adaptation. However, after a shared operator is fine-tuned to multiple regimes within a continuous physical family, it remains unclear whether the resulting weight-space updates merely form isolated regime experts or reveal reusable physical structure. Starting from a shared family anchor, we fine-tune low- and high-regime endpoint experts and show that their updates can be separated into a family-shared adaptation and a direction aligned with the underlying physical parameter. This separation reinterprets endpoint experts as finite-difference probes of a local physical direction in weight space, explaining why static averaging can interpolate between regimes but attenuates endpoint-specific physics. Building on this perspective, we propose Calibration-Conditioned Merge (CCM), a post-hoc coordinate readout method for composing neural PDE experts along this physical direction. Given physical metadata, a calibrated coordinate mapping, or a short observed rollout prefix, CCM infers the target composition coordinate and deploys a single merged checkpoint for the remaining rollout. We evaluate CCM on the reaction--diffusion system, viscosity-parameterized two-dimensional Navier--Stokes equations, and radial dam-break dynamics. Across these benchmarks, CCM achieves its strongest gains in extrapolative regimes, reducing out-of-distribution rollout error relative to the family anchor by 54.2%, 42.8%, and 13.8%, respectively. Further experiments across FNO scales, a DPOT-style backbone, and ablations confirm that endpoint fine-tuning is not arbitrary checkpoint drift, but reveals a calibratable physical direction for training-free transfer across PDE regimes.

Problem

• neural operators
• weight space
• physical direction
• PDE regimes
• fine-tuning

Innovation

• physical direction in weight space
• neural PDE operators
• Calibration-Conditioned Merge (CCM)
• in-context adaptation
• training-free transfer