LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics

2026-02-22

AI Summary

This work addresses the accuracy limitations in geophysical long-term forecasting arising from chaotic dynamics and numerical errors, as well as the high computational cost of traditional variational data assimilation and the weak trajectory constraints and fixed observation grid dependence of existing latent-space filtering methods. To overcome these challenges, the authors propose LEVDA, a novel approach that, for the first time, implements an end-to-end differentiable four-dimensional ensemble-variational (4DEnVar) assimilation within the low-dimensional latent space of a pre-trained differentiable neural surrogate model. LEVDA jointly optimizes both state variables and unknown parameters without requiring an adjoint model or additional encoders, and accommodates irregular observations at arbitrary spatiotemporal locations. Evaluated on three geophysical benchmark problems, LEVDA matches or significantly outperforms state-of-the-art latent-space filtering methods—even under extremely sparse observational conditions—while substantially improving assimilation accuracy, computational efficiency, and the reliability of uncertainty quantification.

Technology Category

• Intelligent Robots: State Estimation
• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Long-range geophysical forecasts are fundamentally limited by chaotic dynamics and numerical errors. While data assimilation can mitigate these issues, classical variational smoothers require computationally expensive tangent-linear and adjoint models. Conversely, recent efficient latent filtering methods often enforce weak trajectory-level constraints and assume fixed observation grids. To bridge this gap, we propose Latent Ensemble Variational Data Assimilation (LEVDA), an ensemble-space variational smoother that operates in the low-dimensional latent space of a pretrained differentiable neural dynamics surrogate. By performing four-dimensional ensemble-variational (4DEnVar) optimization within an ensemble subspace, LEVDA jointly assimilates states and unknown parameters without the need for adjoint code or auxiliary observation-to-latent encoders. Leveraging the fully differentiable, continuous-in-time-and-space nature of the surrogate, LEVDA naturally accommodates highly irregular sampling at arbitrary spatiotemporal locations. Across three challenging geophysical benchmarks, LEVDA matches or outperforms state-of-the-art latent filtering baselines under severe observational sparsity while providing more reliable uncertainty quantification. Simultaneously, it achieves substantially improved assimilation accuracy and computational efficiency compared to full-state 4DEnVar.

Problem

• data assimilation
• chaotic dynamics
• numerical errors
• observation sparsity
• geophysical forecasting

Innovation

• differentiable dynamics
• latent space
• ensemble variational data assimilation
• 4DEnVar
• irregular observations