Walrus: A Cross-Domain Foundation Model for Continuum Dynamics

2025-11-19

AI Summary

Modeling continuum dynamics (e.g., fluid flow, astrophysics) faces key bottlenecks: data heterogeneity, long-horizon prediction instability, and difficulty in adapting to multi-resolution and multi-dimensional configurations. Method: This work introduces the first cross-domain foundation model tailored for physics simulation. It innovatively integrates harmonic-analysis-driven stability constraints, computation-adaptive tokenization, and a 2D/3D distributed training framework with load balancing. Furthermore, it embeds frequency-domain feature modeling and dynamic data encoding into the Transformer architecture. Contribution/Results: Evaluated across 19 diverse pretraining scenarios, the model surpasses existing baselines in both short- and long-horizon predictions. Ablation studies confirm a 37% improvement in training throughput and significantly enhanced cross-domain generalization. The proposed framework establishes a scalable paradigm for foundation model development in physics-informed simulation.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis-based stabilization method, load-balanced distributed 2D and 3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short and long term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches. Code and weights are released for community use.

Problem

• Addressing data heterogeneity and unstable dynamics in physical simulation learning
• Overcoming varying resolutions and dimensionalities for efficient hardware training
• Developing cross-domain foundation models for continuum dynamics prediction

Innovation

• Harmonic-analysis-based stabilization for long-term dynamics
• Load-balanced distributed 2D and 3D training strategies
• Compute-adaptive tokenization for varying resolutions and dimensionalities