Orca: The World is in Your Mind

2026-06-29

AI Summary

This work proposes Orca, a general-purpose world modeling paradigm centered on next-state prediction to unify the understanding, forecasting, and interaction with the real world within a multimodal foundation model. Orca integrates unconscious learning—extracting dense state transitions from continuous video—with conscious learning—modeling sparse semantic transitions via language descriptions and visual question answering (VQA) supervision—to train a unified latent world space on large-scale video and event-annotated data (125K hours of video and 160M events). This latent space enables lightweight, modality-specific decoders that, when trained while keeping the backbone frozen, outperform same-scale specialized models across three distinct tasks: text generation, image prediction, and embodied action generation, thereby demonstrating the framework’s effectiveness and scalability.

Abstract

We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.

Problem

• world foundation model
• unified world latent space
• next-state prediction
• multimodal learning
• general AI

Innovation

• Next-State-Prediction
• world foundation model
• unified latent space
• multimodal learning
• conscious and unconscious learning