Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learning

2026-05-07

AI Summary

This work addresses the systematic degradation of performance in deep neural operators over irregular geometric domains, which arises from the Markovian layer structure and global mixing mechanisms that progressively discard geometric information with increasing depth. The study formally introduces, for the first time, the “geometric forgetting” hypothesis, demonstrating it to be an intrinsic architectural flaw rather than a consequence of insufficient encoding. To mitigate this issue, the authors propose a lightweight geometric memory injection mechanism that reinstates geometric constraints within intermediate layers. Leveraging inter-layer geometric probing and a spectral-attention hybrid architecture, the approach effectively alleviates geometric forgetting without requiring substantial model redesign, yielding significant improvements in accuracy, stability, and generalization on irregular domains. Additionally, the analysis reveals inherent geometric shortcut instabilities in Transformer-based operators.

Abstract

Neural operators perform well on structured domains, yet their behaviour on irregular geometries remains poorly understood. We show that this limitation is not merely an encoding issue, but a depth-wise failure mode inherent to deep operator architectures. We formalise the Geometric Forgetting Hypothesis: due to the Markovian structure of operator layers and their reliance on global mixing mechanisms, neural operators progressively lose access to domain geometry as depth increases. Using layer-wise geometric probing, we demonstrate that both spectral and attention-based operators systematically lose geometric fidelity. We show that this geometric forgetting degrades accuracy, stability, and generalisation. To counteract it, we introduce a lightweight geometry memory injection mechanism that restores geometric constraints at intermediate depths with minimal architectural overhead. This simple intervention consistently mitigates forgetting and exposes a geometric shortcut instability in transformer-based operators, revealing that geometric retention is a structural requirement rather than a design choice.

Problem

• neural operators
• geometric forgetting
• irregular geometries
• depth-wise failure
• geometry retention

Innovation

• Geometric Forgetting
• Neural Operators
• Geometry Memory Injection
• Operator Learning
• Depth-wise Failure