Self-Organizing Score-based Data Assimilation

2026-03-30

AI Summary

This study addresses the challenge of applying Score-based Data Assimilation (SDA) to nonlinear, high-dimensional state-space models where state transitions depend on unknown parameters. To overcome this limitation, the authors introduce a self-organizing mechanism into the SDA framework for the first time, proposing a joint inference approach that integrates diffusion models, score matching, and self-organizing parameter estimation. This novel method effectively circumvents the applicability bottleneck of SDA in systems with unknown parameters, enabling efficient and simultaneous estimation of latent states and model parameters. Experimental results on representative dynamical systems from neuroscience and atmospheric science demonstrate the method’s efficacy and remarkable scalability, as evidenced by its successful application to Kolmogorov flows with state dimensions exceeding one hundred thousand.

Abstract

A state-space model is a statistical framework for inferring latent states from observed time-series data. However, inference with nonlinear and high-dimensional state-space models remains challenging. To this end, an approach based on diffusion models-a powerful class of deep generative models-has been developed, known as Score-based Data Assimilation (SDA). However, SDA cannot be directly applied when the latent-state transition depends on unknown parameters that must be inferred jointly with the latent states. To overcome this limitation, we propose a framework that enables SDA to handle latent states with unknown parameters. A key feature of the proposed method is the incorporation of the self-organization technique, which has been used in classical state-space modeling for the joint estimation of latent states and parameters. By integrating this classical technique into modern SDA, our method enables joint inference of latent states and unknown parameters while maintaining the high training efficiency of SDA. The effectiveness of the proposed approach is validated through numerical experiments on dynamical systems arising in neuroscience and atmospheric science. In addition, its scalability is demonstrated using a high-dimensional Kolmogorov flow, with the data dimension on the order of several hundred thousand.

Problem

• state-space model
• data assimilation
• unknown parameters
• joint inference
• diffusion models

Innovation

• Score-based Data Assimilation
• Self-Organization
• State-Space Models
• Diffusion Models
• Joint Parameter-State Inference