World Machine: Towards Generative World Modeling for Time-Series

2026-05-21

AI Summary

This work addresses the quadratic computational and memory overhead of conventional Transformers with respect to context length, which hinders efficient predictive understanding and controllable simulation in time-series modeling. To overcome this limitation, the authors propose World Machine—a generative world model grounded in a structured latent state mechanism that circumvents the quadratic complexity of standard Transformers and dynamically adapts to both observation volume and context length. The approach integrates a modified Transformer architecture, latent state representation learning, and a tailored training protocol. Evaluated on a custom Toy1D synthetic dataset, World Machine demonstrates superior modeling capacity compared to baseline Transformers, with ablation studies confirming the contribution of each component to overall performance gains.

Abstract

World models represent a paradigm shift in generative AI, pursuing predictive understanding and controllable simulation of environments in a structured and generalizable way. We present World Machine, a generative world-modeling architecture for time series. It is a transformer-based architecture with latent states that enables adaptation to different amounts of observed data and contexts. This shows an improvement over traditional transformers, which have a computational and memory cost that scales quadratically with the context. Experiments on a proposed synthetic dataset, Toy1D, validate the approach's feasibility, demonstrate capabilities not found in conventional transformers, and highlight the contributions of each component of the training protocol.

Problem

• world models
• generative modeling
• time series
• transformer
• latent states

Innovation

• world model
• generative modeling
• time series
• latent state
• efficient transformer