AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

2026-09-13

AI Summary

为解决视频世界模型中场景上下文保持与高保真度观察的问题，AlayaVista通过全景状态到视角视频的转换方法，结合全局上下文和局部细节处理，实现了低延迟的交互式视频生成。

Abstract

Interactive video world models must maintain broad scene context under camera motion while producing high-fidelity observations with low latency. Existing approaches face a representation trade-off: perspective models operate on local views and must preserve off-screen content over long rollouts, whereas broader spatial coverage is typically obtained by synthesizing full-sphere videos or constructing explicit 3D representations. Motivated by the complementary roles of global context and selective local acuity in visual perception, we present AlayaVista, a camera-controllable streaming video world model that decouples panoramic world evolution from perspective observation synthesis. Given a single perspective image, AlayaVista constructs a 360-degree scene prior using a pretrained panorama expansion model and then evolves the scene as a camera-conditioned panoramic latent state. A latent viewport renderer maps this state to the requested perspective video latents, while a perspective refiner restores details, suppresses artifacts, and performs super-resolution. To support efficient streaming, we adapt the panoramic generator to chunk-autoregressive generation and distill both panoramic generation and perspective refinement into few-step processes. To provide the supervision required by this design, we construct MUGEN, a large-scale real-world panoramic video dataset containing 1,318 hours of videos at resolutions of at least 4K, together with rich semantic and geometric annotations.

Problem

• Interactive video world models
• Camera motion
• Scene context
• High-fidelity observations
• Latency

Innovation

• panoramic world model
• perspective video synthesis
• chunk-autoregressive generation
• latent viewport renderer
• super-resolution