AlayaWorld: Interactive Long-Horizon World Modeling - Full Technical Report (v1.1)

2026-08-13

AI Summary

This work addresses the challenge of aligning conditional signals with generated content in spatiotemporal structure for long-horizon interactive world modeling. To this end, it introduces motion-aware latent conditioning, a causally encoded spatial memory sequence, and a hard memory dropping mechanism, while eliminating the camera AdaLN branch and relying entirely on re-rendered spatial conditions to control viewpoint. The method integrates streaming 3D point cache rendering, a causal variational autoencoder (VAE), and chunk-wise autoregressive generation within a unified codec protocol, thereby aligning the temporal statistical properties of visual conditions with those of the generated video. Experiments demonstrate that the proposed framework substantially improves spatiotemporal consistency, long-term coherence, and interactive controllability between generated content and conditioning signals.

Abstract

This report presents an improved version of AlayaWorld. While the backbone architecture, chunk-wise autoregressive generation scheme, and training data remain unchanged from the previous release, we substantially revise how conditioning signals are represented and integrated into the model. The new design is guided by a simple principle: conditioning signals should match the generated content as closely as possible in both latent representation and temporal structure. To this end, we make two major changes. First, we replace the previous depth-warping-based spatial memory with a streaming 3D point-cache renderer. Second, we redesign the conditioning pipeline so that visual conditions are encoded in the same causal-VAE latent space, with temporal statistics consistent with those of the generated video. Concretely, the new version introduces six modifications: (1) replacing static-frame image conditioning with motion-aware latent conditioning; (2) causally encoding re-rendered spatial memory as a continuous sequence; (3) aligning the temporal-memory window in pixel space; (4) adopting hard memory dropout that removes memory tokens rather than zeroing them; (5) unifying the VAE encoding and decoding protocol across training and inference; and (6) removing the camera AdaLN branch, such that viewpoint control is provided entirely through the re-rendered spatial condition.

Problem

• world modeling
• long-horizon generation
• conditioning signals
• temporal structure
• latent representation

Innovation

• causal-VAE
• 3D point-cache renderer
• motion-aware conditioning
• temporal alignment
• hard memory dropout