DAISI: Data Assimilation with Inverse Sampling using Stochastic Interpolants

2025-11-28

AI Summary

Traditional data assimilation (DA) methods—such as the ensemble Kalman filter—rely on Gaussian assumptions and heuristic parameter tuning, leading to instability or substantial bias in non-Gaussian, strongly nonlinear systems. To address these limitations, we propose DAISI, a scalable DA framework based on normalizing flow generative models. DAISI maps forecast ensembles into a latent space via invertible sampling, embeds dynamical priors into a pre-trained generative model without fine-tuning, and fuses forecasts with observations through latent-space conditioning. It further constructs a data-driven prior using stochastic interpolation within the latent space, enabling efficient probabilistic inference. Experiments demonstrate that DAISI significantly outperforms conventional DA methods in high-dimensional, non-Gaussian, sparsely observed, and strongly nonlinear settings—achieving both superior filtering accuracy and numerical stability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Data assimilation (DA) is a cornerstone of scientific and engineering applications, combining model forecasts with sparse and noisy observations to estimate latent system states. Classical DA methods, such as the ensemble Kalman filter, rely on Gaussian approximations and heuristic tuning (e.g., inflation and localization) to scale to high dimensions. While often successful, these approximations can make the methods unstable or inaccurate when the underlying distributions of states and observations depart significantly from Gaussianity. To address this limitation, we introduce DAISI, a scalable filtering algorithm built on flow-based generative models that enables flexible probabilistic inference using data-driven priors. The core idea is to use a stationary, pre-trained generative prior to assimilate observations via guidance-based conditional sampling while incorporating forecast information through a novel inverse-sampling step. This step maps the forecast ensemble into a latent space to provide initial conditions for the conditional sampling, allowing us to encode model dynamics into the DA pipeline without having to retrain or fine-tune the generative prior at each assimilation step. Experiments on challenging nonlinear systems show that DAISI achieves accurate filtering results in regimes with sparse, noisy, and nonlinear observations where traditional methods struggle.

Problem

• Develops a scalable filtering algorithm for non-Gaussian data assimilation
• Uses flow-based generative models to incorporate data-driven priors
• Enables accurate state estimation with sparse, noisy nonlinear observations

Innovation

• Flow-based generative models for flexible probabilistic inference
• Inverse-sampling step mapping forecast ensemble into latent space
• Guidance-based conditional sampling with stationary pre-trained generative prior