From Masks to Worlds: A Hitchhiker's Guide to World Models

2025-10-23

AI Summary

This paper addresses three fundamental challenges in real-world modeling: (1) achieving cross-modal unified representation, (2) enabling closed-loop perception–action interaction, and (3) maintaining long-term dynamic consistency. To this end, it proposes an evolutionary paradigm grounded in three pillars: generativity, interactivity, and memory. Methodologically, it departs from generic survey-based approaches and instead designs an end-to-end system integrating cross-modal representation learning, a unified generative architecture, closed-loop perception–action mechanisms, and continual memory-augmented neural networks—evolving from early masked modeling toward interactive, temporally consistent generative world models. Its core contribution is the first systematic distillation of the key technical pathway toward sustainable, interactive world models; it establishes the foundational paradigm for dynamic virtual world modeling and provides a reproducible framework and practical guidelines for both the theoretical development and engineering implementation of embodied agents.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This is not a typical survey of world models; it is a guide for those who want to build worlds. We do not aim to catalog every paper that has ever mentioned a ``world model". Instead, we follow one clear road: from early masked models that unified representation learning across modalities, to unified architectures that share a single paradigm, then to interactive generative models that close the action-perception loop, and finally to memory-augmented systems that sustain consistent worlds over time. We bypass loosely related branches to focus on the core: the generative heart, the interactive loop, and the memory system. We show that this is the most promising path towards true world models.

Problem

• Building generative world models from masked representations
• Developing interactive models with action-perception loops
• Creating memory-augmented systems for consistent worlds

Innovation

• Masked models unify multimodal representation learning
• Unified architectures share single paradigm across tasks
• Memory-augmented systems sustain consistent temporal worlds