UnHiPPO: Uncertainty-aware Initialization for State Space Models

2025-06-05

AI Summary

State space models (SSMs) commonly employ the HiPPO framework for dynamic initialization, yet HiPPO assumes noise-free data—a strong idealization that undermines robustness on real-world noisy sequences. Method: We reinterpret HiPPO as a linear stochastic control problem and, for the first time, reformulate it as Bayesian inference in a latent-variable system with measurement noise. Building on this insight, we propose an uncertainty-aware initialization method that jointly estimates initial states and parameters via Bayesian filtering—without increasing inference overhead. Contribution/Results: Our approach unifies HiPPO’s projection theory, stochastic control, and Bayesian inference into a coherent framework. It significantly improves training stability and inference robustness of SSMs under noise, establishing a new paradigm for designing noise-resilient SSMs. Experimental results demonstrate consistent gains across diverse noisy time-series benchmarks, validating both theoretical soundness and practical efficacy.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation
• Machine Learning: Bayesian Learning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time. Find our implementation at https://cs.cit.tum.de/daml/unhippo.

Problem

• Extends HiPPO theory to handle measurement noise in data
• Derives uncertainty-aware initialization for state space models
• Improves noise resistance in training and inference phases

Innovation

• Extends HiPPO with measurement noise theory
• Reformulates data as noisy outputs of latent system
• Infers latent system posterior without runtime increase