Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems

2026-02-01

AI Summary

This work addresses the challenge of spectral bias in machine learning models for dynamical system modeling, which often leads to inaccurate representation of high-frequency components and instability in long-term predictions. To mitigate this issue, the authors propose the Multi-Scale Wavelet Transformer (MSWT), a novel operator learning framework that operates in the wavelet domain. By integrating wavelet-preserving downsampling and cross-scale wavelet attention mechanisms, MSWT explicitly separates and retains both high- and low-frequency information. This approach effectively alleviates spectral bias and significantly enhances the model’s capacity to capture high-frequency structures. Extensive experiments on chaotic dynamical systems and ERA5 climate reanalysis data demonstrate substantial reductions in prediction error and climatic bias, yielding improved long-term stability and spectral fidelity.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Multiagent Systems: Multiagent Learning
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent years have seen a surge in data-driven surrogates for dynamical systems that can be orders of magnitude faster than numerical solvers. However, many machine learning-based models such as neural operators exhibit spectral bias, attenuating high-frequency components that often encode small-scale structure. This limitation is particularly damaging in applications such as weather forecasting, where misrepresented high frequencies can induce long-horizon instability. To address this issue, we propose multi-scale wavelet transformers (MSWTs), which learn system dynamics in a tokenized wavelet domain. The wavelet transform explicitly separates low- and high-frequency content across scales. MSWTs leverage a wavelet-preserving downsampling scheme that retains high-frequency features and employ wavelet-based attention to capture dependencies across scales and frequency bands. Experiments on chaotic dynamical systems show substantial error reductions and improved long horizon spectral fidelity. On the ERA5 climate reanalysis, MSWTs further reduce climatological bias, demonstrating their effectiveness in a real-world forecasting setting.

Problem

• spectral bias
• high-frequency components
• dynamical systems
• long-horizon instability
• neural operators

Innovation

• wavelet transform
• neural operators
• spectral bias
• multi-scale modeling
• dynamical systems