MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction

2026-03-16

AI Summary

This work addresses the persistent performance degradation in existing streaming 3D reconstruction methods caused by state drift and catastrophic forgetting over long sequences. To mitigate this, we propose MeMix—a training-free, plug-and-play memory mixing module that restructures recurrent states into a memory ensemble and selectively updates only the least-aligned memory component while preserving the rest with high fidelity. MeMix introduces, for the first time, a selective memory update mechanism that operates without increasing inference memory beyond O(1), adding no extra parameters, or requiring fine-tuning, enabling seamless integration into existing architectures. Evaluated on standard benchmarks including ScanNet, 7-Scenes, and KITTI, MeMix reduces average reconstruction completeness error by 15.3% under identical backbone networks, with improvements reaching up to 40.0%.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reconstruction is a fundamental task in 3D vision and a fundamental capability for spatial intelligence. Particularly, streaming 3D reconstruction is central to real-time spatial perception, yet existing recurrent online models often suffer from progressive degradation on long sequences due to state drift and forgetting, motivating inference-time remedies. We present MeMix, a training-free, plug-and-play module that improves streaming reconstruction by recasting the recurrent state into a Memory Mixture. MeMix partitions the state into multiple independent memory patches and updates only the least-aligned memory patches while exactly preserving others. This selective update mitigates catastrophic forgetting while retaining $O(1)$ inference memory, and requires no fine-tuning or additional learnable parameters, making it directly applicable to existing recurrent reconstruction models. Across standard benchmarks (ScanNet, 7-Scenes, KITTI, etc.), under identical backbones and inference settings, MeMix reduces reconstruction completeness error by 15.3% on average (up to 40.0%) across 300--500 frame streams on 7-Scenes. The code is available at https://dongjiacheng06.github.io/MeMix/

Problem

• streaming 3D reconstruction
• state drift
• catastrophic forgetting
• recurrent models
• spatial perception

Innovation

• MeMix
• streaming 3D reconstruction
• memory mixture
• catastrophic forgetting
• plug-and-play module