Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning

2026-05-14

AI Summary

This work proposes a self-supervised autoencoder-based foundation model for three-dimensional partial differential equations (PDEs) to address key challenges including poor transferability, difficulty in scaling to high dimensions, and limited functional versatility. By integrating online synthetic data generation, low-rank adaptation fine-tuning, latent-space dynamics modeling, and enhanced skip connections, the model is efficiently pre-trained on hundreds of terabytes of equivalent data to learn universal representations across diverse physical systems. The resulting framework supports multiple downstream tasks—such as dynamical prediction and generative modeling—and demonstrates strong performance across a range of PDE problems after fine-tuning, significantly advancing scalability, generalizability, and multifunctionality in scientific machine learning.

Abstract

We introduce Tadpole, a novel foundation model for three-dimensional partial differential equations (PDEs) that addresses key challenges in transferability, scalability to high dimensionality, and multi-functionality. Tadpole is pre-trained as an autoencoder on synthetic 3D PDE data generated by an efficient online data-generation framework. This enables large-scale, diverse training without storage or I/O overhead, demonstrated by scaling to an equivalent of hundreds of terabytes of training data. By autoencoding single-channel spatial crops, Tadpole learns rich and transferable representations across heterogeneous physical systems with varying numbers of state variables and spatial resolutions. Although pre-trained solely as an autoencoder, Tadpole can be efficiently applied for multiple downstream tasks beyond reconstruction, including dynamics learning and generative modeling. For dynamics learning, we propose a novel parameter-efficient fine-tuning strategy that integrates low-rank adaptation, latent-space transformations, and reintroduced skip connections, achieving accurate temporal modeling with a minimal number of trainable parameters. Tadpole demonstrates strong fine-tuning performance across various downstream tasks, highlighting its versatility and effectiveness as a foundation model for 3D PDE learning. Source code and pre-trained weights of Tadpole are available at https://github.com/tum-pbs/tadpole

Problem

• 3D PDEs
• transferability
• scalability
• multi-functionality
• foundation models

Innovation

• foundation model
• autoencoder
• 3D PDEs
• online learning
• parameter-efficient fine-tuning