A Continuous-Time Ensemble Kalman-Bucy Smoother for Causal Inference and Model Discovery

2026-04-27

AI Summary

This study addresses the latency and bias in state estimation inherent to conventional filtering methods when applied to rapidly evolving or regime-switching complex systems, which rely solely on past and current observations. To overcome this limitation, the authors propose a continuous-time ensemble Kalman–Bucy smoother (EnKBS) that incorporates future observations to reconstruct the conditional distribution, enabling high-accuracy retrospective state estimation and facilitating causal inference and implicit model structure discovery. The method innovatively establishes, for the first time, a continuous-time smoothing framework that requires neither tangent linear nor adjoint models and converges to the exact solution in the infinite-ensemble limit. By integrating ensemble-based moment estimation with regularization techniques such as covariance localization and inflation, it avoids explicit derivative computations. With only O(10) ensemble members and partial observations, the approach successfully infers causal dynamics in a bivariate trigger-feedback system and recovers structure and parameters in a simplified atmospheric circulation model, substantially outperforming traditional filters.

Abstract

Data assimilation (DA) integrates observational information with model predictions to improve state estimation in complex systems. While filtering provides the basis for online forecasts by using only past and present observations, it can exhibit delays and biases when the underlying dynamics evolve rapidly or undergo regime transitions. Smoothing, which additionally incorporates future observations, provides a natural pipeline for hindcasting and reanalysis that yields an uncertainty reduction beyond the filter. This paper introduces an ensemble Kalman-Bucy smoother (EnKBS) for continuous-time DA of nonlinear dynamical systems, where the smoother's conditional distributions are reconstructed using ensemble moments. The result is a derivative-free framework that does not require explicit computation of tangent-linear or adjoint models, which converges to the exact smoother solution at the infinite-ensemble limit for a wide class of complex systems. Incorporating standard regularization techniques for high-dimensional systems, such as covariance localization and inflation, the skill of the EnKBS is demonstrated in various important scientific problems. By integrating future observations, which reveal the underlying causal mechanisms for retrospective state updates, the EnKBS is used for Bayesian-based inference of causal relationships and their temporal influence range in a dyadic trigger-feedback model and the development of a causality-driven iterative learning algorithm that identifies the structure and recovers the hidden parameters of a nonlinear reduced-order model mimicking midlatitude atmospheric circulation. Notably, both tasks remain effective with an ensemble size of $O(10)$ under partial observations, suggesting that EnKBS can support the instantaneous discovery of high-dimensional complex systems over time.

Problem

• causal inference
• model discovery
• data assimilation
• smoothing
• nonlinear dynamical systems

Innovation

• ensemble Kalman-Bucy smoother
• continuous-time data assimilation
• causal inference
• model discovery
• nonlinear dynamical systems