In-Context Multi-Operator Learning with DeepOSets

2025-12-17

AI Summary

Existing operator learning methods struggle with zero-shot generalization to unseen partial differential equations (PDEs) in multi-operator contextual learning settings, typically requiring fine-tuning or task-specific architectures. Method: We propose DeepOSets—a zero-weight-update framework integrating DeepSets and DeepONets—to enable instantaneous generalization to unseen PDE solution operators. It introduces an operator-level context encoding mechanism and employs a non-autoregressive, attention-free inference paradigm. Contribution/Results: We theoretically prove that DeepOSets can uniformly approximate a class of continuous operators. Experiments on forward and inverse problems for Poisson and reaction-diffusion equations demonstrate high-accuracy prediction of previously unencountered PDE solution operators using only a few parameter–solution example pairs as prompts. To our knowledge, this is the first work achieving in-context learning across distinct PDEs and multiple operator families without weight updates—effectively bypassing conventional fine-tuning or architectural specialization constraints.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In-context Learning (ICL) is the remarkable capability displayed by some machine learning models to learn from examples in a prompt, without any further weight updates. ICL had originally been thought to emerge from the self-attention mechanism in autoregressive transformer architectures. DeepOSets is a non-autoregressive, non-attention based neural architecture that combines set learning via the DeepSets architecture with operator learning via Deep Operator Networks (DeepONets). In a previous study, DeepOSets was shown to display ICL capabilities in supervised learning problems. In this paper, we show that the DeepOSets architecture, with the appropriate modifications, is a multi-operator in-context learner that can recover the solution operator of a new PDE, not seen during training, from example pairs of parameter and solution placed in a user prompt, without any weight updates. Furthermore, we show that DeepOSets is a universal uniform approximator over a class of continuous operators, which we believe is the first result of its kind in the literature of scientific machine learning. This means that a single DeepOSets architecture exists that approximates in-context any continuous operator in the class to any fixed desired degree accuracy, given an appropriate number of examples in the prompt. Experiments with Poisson and reaction-diffusion forward and inverse boundary-value problems demonstrate the ability of the proposed model to use in-context examples to predict accurately the solutions corresponding to parameter queries for PDEs not seen during training.

Problem

• DeepOSets learns PDE solution operators from in-context examples without weight updates.
• It approximates continuous operators universally for scientific machine learning tasks.
• The model predicts solutions for unseen PDEs using example pairs in prompts.

Innovation

• DeepOSets combines DeepSets with DeepONets for operator learning
• It learns PDE solution operators in-context without weight updates
• It is a universal approximator for continuous operators via prompts