FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction

2026-03-19

AI Summary

This work addresses the challenge in streaming 3D reconstruction where existing latent state update strategies struggle to balance retention of historical information with responsiveness to new observations, leading to unstable long-term inference. The authors propose FILT3R—a training-free, interpretable, and plug-and-play recursive update rule that formulates state updates as a stochastic estimation problem in token space. Inspired by Kalman filtering, FILT3R maintains per-token variance to compute adaptive gains and incorporates an EMA-normalized temporal drift estimator to online approximate process noise. This framework unifies existing coverage- and gating-based strategies as special cases. Experiments demonstrate that FILT3R significantly improves long-horizon stability in depth estimation, pose tracking, and 3D reconstruction, outperforming current methods.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Streaming 3D reconstruction maintains a persistent latent state that is updated online from incoming frames, enabling constant-memory inference. A key failure mode is the state update rule: aggressive overwrites forget useful history, while conservative updates fail to track new evidence, and both behaviors become unstable beyond the training horizon. To address this challenge, we propose FILT3R, a training-free latent filtering layer that casts recurrent state updates as stochastic state estimation in token space. FILT3R maintains a per-token variance and computes a Kalman-style gain that adaptively balances memory retention against new observations. Process noise -- governing how much the latent state is expected to change between frames -- is estimated online from EMA-normalized temporal drift of candidate tokens. Using extensive experiments, we demonstrate that FILT3R yields an interpretable, plug-in update rule that generalizes common overwrite and gating policies as special cases. Specifically, we show that gains shrink in stable regimes as uncertainty contracts with accumulated evidence, and rise when genuine scene change increases process uncertainty, improving long-horizon stability for depth, pose, and 3D reconstruction, compared to the existing methods. Code will be released at https://github.com/jinotter3/FILT3R.

Problem

• streaming 3D reconstruction
• latent state update
• online inference
• temporal stability
• state estimation

Innovation

• Kalman filter
• streaming 3D reconstruction
• latent state estimation
• adaptive filtering
• token-level uncertainty