Astra: General Interactive World Model with Autoregressive Denoising

2025-12-09

AI Summary

Existing world models struggle to achieve high-fidelity, long-horizon future video prediction under joint multimodal observation (e.g., vision, language) and heterogeneous action inputs (e.g., robotic manipulation, camera motion) in general-purpose settings. This paper introduces the first general-purpose interactive world model, built upon an autoregressive diffusion Transformer architecture. Key innovations include time-causal attention, noise-augmented historical memory, action-aware adapters, and a Mixture-of-Action-Experts (MoAE) routing mechanism—collectively balancing responsiveness and temporal consistency. The model autoregressively synthesizes high-fidelity, long-horizon video futures conditioned on multimodal history and diverse real-time action signals. It achieves significant improvements across benchmarks in prediction fidelity, long-range temporal coherence, and action-dynamics alignment. To our knowledge, this is the first work enabling general-purpose interactive world modeling for real-world applications such as autonomous driving and dexterous manipulation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Recent advances in diffusion transformers have empowered video generation models to generate high-quality video clips from texts or images. However, world models with the ability to predict long-horizon futures from past observations and actions remain underexplored, especially for general-purpose scenarios and various forms of actions. To bridge this gap, we introduce Astra, an interactive general world model that generates real-world futures for diverse scenarios (e.g., autonomous driving, robot grasping) with precise action interactions (e.g., camera motion, robot action). We propose an autoregressive denoising architecture and use temporal causal attention to aggregate past observations and support streaming outputs. We use a noise-augmented history memory to avoid over-reliance on past frames to balance responsiveness with temporal coherence. For precise action control, we introduce an action-aware adapter that directly injects action signals into the denoising process. We further develop a mixture of action experts that dynamically route heterogeneous action modalities, enhancing versatility across diverse real-world tasks such as exploration, manipulation, and camera control. Astra achieves interactive, consistent, and general long-term video prediction and supports various forms of interactions. Experiments across multiple datasets demonstrate the improvements of Astra in fidelity, long-range prediction, and action alignment over existing state-of-the-art world models.

Problem

• Predicts long-horizon futures from observations and actions
• Generates real-world video for diverse general-purpose scenarios
• Enables precise control with various forms of action interactions

Innovation

• Autoregressive denoising with temporal causal attention
• Noise-augmented history memory for temporal coherence
• Action-aware adapter and mixture of experts for control