Walking in the Implicit: Interactive World Exploration via Neural Scene Representation

2026-06-29

AI Summary

Existing interactive video generation methods struggle to simultaneously achieve long-term temporal consistency and efficient rendering due to the tight coupling between state transitions and high-frequency observation synthesis. This work proposes a Neural Implicit Scene (NIS) representation paradigm, which replaces per-frame latent variables with a fixed-length, renderable implicit state, thereby decoupling world exploration into stochastic transitions over a compact scene state and pose-conditioned rendering. A unified VAE encoder integrates camera poses, reference images, and historical context, while a Transformer-based VAE learns locally anchored NIS representations. Temporal evolution of the NIS is governed by a diffusion Transformer conditioned on future camera trajectories and geometry-aware history. Trained from scratch on public multi-view datasets without relying on pretrained video backbones or explicit 3D reconstruction modules, the model achieves strong temporal coherence and efficient inference.

Abstract

Interactive video generation systems for camera-controlled world exploration roll out growing sequences of latent video frames, entangling state transition with high-frequency observation synthesis. We propose Walking in the Implicit, a scene-centric paradigm that changes the rollout variable from frame latents to a fixed-length, renderable implicit state, termed Neural Implicit Scene (NIS). This factorizes interactive generation into stochastic transition of a compact scene state and deterministic pose-conditioned rendering given the sampled state. We instantiate this paradigm as NeuWorld: a transformer VAE learns locally anchored NIS from sparse posed frames, and a diffusion transformer evolves NIS conditioned on future camera trajectories and geometry-aware retrieved history. By reusing the VAE encoder as a unified conditioner, NeuWorld maps camera, reference-image, and history cues into the same NIS modality, avoiding external heterogeneous encoders. Trained from scratch on public posed-view data without pretrained video backbones or auxiliary 3D reconstructors, NeuWorld achieves strong long-horizon consistency with favorable inference efficiency.

Problem

• interactive video generation
• world exploration
• scene representation
• state transition
• long-horizon consistency

Innovation

• Neural Implicit Scene
• Interactive Video Generation
• Transformer VAE
• Diffusion Transformer
• Pose-Conditioned Rendering