NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos

2026-01-01

arXiv.org

AI Summary

This work proposes a scalable 4D world model that enables high-quality 4D reconstruction and novel viewpoint trajectory generation from in-the-wild monocular videos, circumventing the reliance on costly multi-view data or complex preprocessing inherent in existing approaches. The method introduces three key innovations: a feedforward 4D reconstruction architecture that operates without explicit pose estimation, an online simulation mechanism for monocular degeneracy patterns, and an end-to-end training strategy tailored for unconstrained real-world video. Evaluated on standard benchmarks for 4D reconstruction and video generation, the approach achieves state-of-the-art performance and demonstrates broad applicability across diverse downstream tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

In this paper, we propose NeoVerse, a versatile 4D world model that is capable of 4D reconstruction, novel-trajectory video generation, and rich downstream applications. We first identify a common limitation of scalability in current 4D world modeling methods, caused either by expensive and specialized multi-view 4D data or by cumbersome training pre-processing. In contrast, our NeoVerse is built upon a core philosophy that makes the full pipeline scalable to diverse in-the-wild monocular videos. Specifically, NeoVerse features pose-free feed-forward 4D reconstruction, online monocular degradation pattern simulation, and other well-aligned techniques. These designs empower NeoVerse with versatility and generalization to various domains. Meanwhile, NeoVerse achieves state-of-the-art performance in standard reconstruction and generation benchmarks. Our project page is available at https://neoverse-4d.github.io

Problem

• 4D world model
• scalability
• in-the-wild monocular videos
• 4D reconstruction
• multi-view data

Innovation

• 4D world model
• monocular video
• pose-free reconstruction
• scalable modeling
• novel-trajectory generation