Adaptive Learned State Estimation based on KalmanNet

2026-04-02

AI Summary

This work addresses the challenge of effectively modeling heterogeneous noise characteristics across multiple sensors—radar, lidar, and cameras—in complex traffic scenarios, a limitation of existing hybrid state estimation algorithms. To this end, we propose Adaptive Multimodal KalmanNet (AM-KNet), which synergistically integrates model-driven and data-driven paradigms. AM-KNet incorporates sensor-specific measurement modules, a context-modulated hypernetwork conditioned on object type and motion state, a Joseph-form covariance update branch for numerical stability, and a component-wise loss function that embeds physical priors. Experimental evaluation on the nuScenes and View-of-Delft real-world autonomous driving datasets demonstrates that AM-KNet significantly improves state estimation accuracy and tracking robustness, substantially narrowing the performance gap with classical Bayesian filters.

Technology Category

• Intelligent Robots: State Estimation
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Hybrid state estimators that combine model-based Kalman filtering with learned components have shown promise on simulated data, yet their performance on real-world automotive data remains insufficient. In this work we present Adaptive Multi-modal KalmanNet (AM-KNet), an advancement of KalmanNet tailored to the multi-sensor autonomous driving setting. AM-KNet introduces sensor-specific measurement modules that enable the network to learn the distinct noise characteristics of radar, lidar, and camera independently. A hypernetwork with context modulation conditions the filter on target type, motion state, and relative pose, allowing adaptation to diverse traffic scenarios. We further incorporate a covariance estimation branch based on the Josephs form and supervise it through negative log-likelihood losses on both the estimation error and the innovation. A comprehensive, component-wise loss function encodes physical priors on sensor reliability, target class, motion state, and measurement flow consistency. AM-KNet is trained and evaluated on the nuScenes and View-of-Delft datasets. The results demonstrate improved estimation accuracy and tracking stability compared to the base KalmanNet, narrowing the performance gap with classical Bayesian filters on real-world automotive data.

Problem

• state estimation
• autonomous driving
• multi-sensor fusion
• real-world data
• Kalman filtering

Innovation

• Adaptive KalmanNet
• multi-sensor fusion
• hypernetwork with context modulation
• covariance estimation
• physics-informed loss