Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms

2026-03-21

AI Summary

This work addresses the limitations of traditional data assimilation methods, which rely on precise dynamical models and require laborious parameter tuning, often underperforming under partial and noisy observations. The authors propose a differentiable data assimilation framework that, for the first time, enables end-to-end joint learning of system states, dynamical models, and filtering parameters. Built upon automatic differentiation and gradient-based optimization, the approach incorporates theory-informed loss functions and is compatible with diverse observation models and computational constraints. Validation across dynamical systems in aerospace, atmospheric, and biological domains demonstrates that the framework not only reproduces classical assimilation results but also achieves markedly improved generalization and adaptability.

Technology Category

• Intelligent Robots: State Estimation
• Search and Optimization: Learning to Search
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Data assimilation algorithms estimate the state of a dynamical system from partial observations, where the successful performance of these algorithms hinges on costly parameter tuning and on employing an accurate model for the dynamics. This paper introduces a framework for jointly learning the state, dynamics, and parameters of filtering algorithms in data assimilation through a process we refer to as auto-differentiable filtering. The framework leverages a theoretically motivated loss function that enables learning from partial, noisy observations via gradient-based optimization using auto-differentiation. We further demonstrate how several well-known data assimilation methods can be learned or tuned within this framework. To underscore the versatility of auto-differentiable filtering, we perform experiments on dynamical systems spanning multiple scientific domains, such as the Clohessy-Wiltshire equations from aerospace engineering, the Lorenz-96 system from atmospheric science, and the generalized Lotka-Volterra equations from systems biology. Finally, we provide guidelines for practitioners to customize our framework according to their observation model, accuracy requirements, and computational budget.

Problem

• data assimilation
• state estimation
• dynamical systems
• parameter tuning
• filtering algorithms

Innovation

• auto-differentiable filtering
• data assimilation
• joint learning
• gradient-based optimization
• dynamical systems