Online Neural Space Time Memory for Dynamic Novel View Synthesis

2026-07-16

AI Summary

This work addresses the challenge of achieving real-time dynamic novel view synthesis in multi-view streaming video, which demands both long-term temporal consistency and strict real-time performance. The authors propose an online synthesis framework based on test-time training (TTT) that decouples memory update from application frequency: scene memory is updated only periodically, while each frame efficiently reuses existing memory. To handle dynamic deformations, cross-view attention is introduced. Memory quality is preserved through a dedicated memory loss that enforces internalization of scene structure and a memory caching strategy that mitigates weight drift. The method achieves minute-scale online memory adaptation and real-time rendering in scenes with complex human motion, setting a new state-of-the-art in performance.

Abstract

Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts. Given that memory updates are more demanding than memory application and video content is largely redundant, we propose to decouple the frequencies of these two processes. Our approach performs periodic memory updates while applying the memory on a per-frame basis, using cross-view attention to manage deformations between the prior memory state and the current frame. To lock in the historical context, we introduce two critical mechanisms: an auxiliary Memory Loss that forces persistent internalization of the scene, and a Memory Caching strategy that regularizes active weights against catastrophic drift. Our method demonstrates real-time, state-of-the-art performance on scenes with dynamic human motion as well as minute-scale online memorization.

Problem

• novel view synthesis
• dynamic scenes
• real-time constraints
• long-horizon memory
• occlusion reconstruction

Innovation

• Online Novel View Synthesis
• Neural Space-Time Memory
• Test-Time Training
• Memory Decoupling
• Dynamic Scene Reconstruction