Learning to be Smooth: An End-to-End Differentiable Particle Smoother

2025-02-14

AI Summary

This paper addresses the offline analysis challenge of multimodal state estimation in vision and robotics, where conventional particle smoothers rely on hand-crafted dynamical and observation models and struggle with multi-modal posteriors. We propose an end-to-end differentiable forward–backward particle smoothing framework. Our key contributions are: (1) the first method enabling low-variance gradient propagation over long sequences; (2) a hierarchical resampling and importance-weighting mechanism ensuring differentiability and stability of dual particle streams (forward and backward); and (3) joint learning of neural dynamical and observation models. Evaluated on city-scale video-to-map vehicle global localization—a task demanding robust multimodal temporal reasoning—our approach significantly outperforms state-of-the-art particle filters and search-based baselines. Results demonstrate its effectiveness in handling complex, multi-modal sequential inference under realistic large-scale conditions.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

For challenging state estimation problems arising in domains like vision and robotics, particle-based representations attractively enable temporal reasoning about multiple posterior modes. Particle smoothers offer the potential for more accurate offline data analysis by propagating information both forward and backward in time, but have classically required human-engineered dynamics and observation models. Extending recent advances in discriminative training of particle filters, we develop a framework for low-variance propagation of gradients across long time sequences when training particle smoothers. Our"two-filter'' smoother integrates particle streams that are propagated forward and backward in time, while incorporating stratification and importance weights in the resampling step to provide low-variance gradient estimates for neural network dynamics and observation models. The resulting mixture density particle smoother is substantially more accurate than state-of-the-art particle filters, as well as search-based baselines, for city-scale global vehicle localization from real-world videos and maps.

Problem

• Develops differentiable particle smoother
• Improves state estimation accuracy
• Enables neural network dynamics integration

Innovation

• Differentiable particle smoother
• Two-filter time propagation
• Low-variance gradient estimation