HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

2026-04-15

AI Summary

This work addresses the challenge of efficiently generating high-fidelity, navigable 3D scenes from multimodal inputs—including text, single images, multi-view images, or videos—by introducing a four-stage pipeline: panoramic generation, trajectory planning, world expansion, and synthesis. The framework integrates several advanced components: HY-Pano 2.0, WorldNav, WorldStereo 2.0, WorldMirror 2.0, and the high-performance rendering platform WorldLens. Notably, WorldStereo 2.0 employs a keyframe-based view synthesis mechanism with consistent memory, while WorldMirror 2.0 serves as a general-purpose 3D prediction model. Built upon 3D Gaussian Splatting, the entire system achieves state-of-the-art performance on multiple open-source benchmarks, matching the quality of the closed-source model Marble. The authors have publicly released all code, model weights, and technical details.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.

Problem

• 3D world generation
• multi-modal input
• 3D reconstruction
• world simulation
• 3D Gaussian Splatting

Innovation

• multi-modal world model
• 3D Gaussian Splatting
• panorama generation
• consistent view synthesis
• interactive 3D rendering