omniverse

This work addresses the challenge that 2D panoramic images inherently lack the geometric and material fidelity required for physically based rendering (PBR), relighting, and simulation. To this end, we propose the first PBR-oriented panoramic-driven 3D scene generation framework. Our method jointly models panoramic geometry, texture, and PBR-compliant material properties—including albedo, roughness, and metallic—within a unified representation. A lightweight cross-modal adapter bridges pretrained 2D generative priors with panoramic understanding tasks, while a large-scale, multimodal synthetic panoramic dataset enables robust training. Unlike prior approaches focused solely on appearance reconstruction, ours is the first to achieve end-to-end, perception-guided generation of PBR-ready 3D scenes. Experiments demonstrate significant improvements over state-of-the-art methods on panoramic completion and 3D reconstruction. The generated scenes are directly compatible with real-time PBR rendering and virtual simulation, establishing a new paradigm for immersive virtual world construction.

Current 4D world modeling is hindered by the scarcity of high-quality, highly dynamic, and multi-domain data. Existing benchmarks suffer from limited spatiotemporal complexity, insufficient modality diversity, and inadequate support for key tasks—including 4D geometric reconstruction, future action prediction, and camera-controllable video generation. To address this, we introduce OmniWorld: the first large-scale, multi-domain, multimodal 4D world modeling dataset. It features a newly collected, interaction-rich, photorealistic sub-dataset—OmniWorld-Game—with fine-grained spatiotemporal annotations. Leveraging multi-source acquisition, cross-modal alignment, and collaborative fine-tuning with generative models, we establish a unified benchmark. Experiments demonstrate substantial improvements over state-of-the-art methods in both 4D reconstruction and video generation, while enabling rigorous cross-domain evaluation on high-stakes tasks. Our work validates the critical role of data-driven paradigms in advancing general-purpose 4D understanding.

Existing 3D generation methods often neglect physical properties or are confined to a single object category, failing to meet the demand for diverse and physically plausible assets in downstream simulation tasks. This work proposes PhysX-Omni, a unified framework that achieves joint generation of rigid, deformable, and articulated 3D objects with physical fidelity for the first time. Key innovations include a compression-free, high-resolution geometric representation tailored for vision-language models, the first universal simulation-ready 3D dataset—PhysXVerse—and PhysX-Bench, a comprehensive evaluation benchmark encompassing six-dimensional physical attributes. Experiments demonstrate that the proposed method excels on both conventional metrics and PhysX-Bench, significantly enhancing performance in downstream applications such as simulation scene generation and robot policy learning.

This work addresses the fragmentation between multi-view image-driven 3D scene understanding and generation. We propose a unified “generation-empowered understanding” framework. Methodologically, we design a dual-module architecture—texture and geometry—that operate synergistically: the texture module models spatiotemporal consistency to enable high-fidelity novel-view synthesis, while the geometry module incorporates explicit structural constraints to improve depth and surface normal estimation accuracy; a two-stage training strategy enables bidirectional optimization. Our key innovation lies in the first use of generative tasks (e.g., rendering) as supervisory signals to enhance 3D understanding—reversing the conventional unidirectional paradigm. On the VSI-Bench benchmark, our method achieves a state-of-the-art score of 55.4, while significantly outperforming prior approaches on both novel-view synthesis and geometric estimation tasks.

Metaverse applications face critical bottlenecks including high real-time rendering latency, poor adaptability to dynamic scenes, and limited scalability. To address these challenges, this paper proposes a large language model (LLM)-empowered cloud-edge-device collaborative generative AI rendering framework. It introduces two key innovations: (1) a mobility-aware pre-rendering mechanism that anticipates user movement for proactive resource allocation, and (2) a diffusion model–driven adaptive rendering strategy that dynamically optimizes visual fidelity and computational load based on scene complexity and device capabilities. The framework tightly integrates LLMs, video foundation models (e.g., Sora), and hierarchical distributed computing across cloud, edge, and end devices. Experimental evaluation demonstrates a 37% reduction in end-to-end rendering latency and significantly enhanced real-time immersion under high-concurrency, highly dynamic conditions. This work establishes a scalable, generative-AI-native technical pathway for next-generation metaverse systems.

Existing world model evaluation benchmarks predominantly emphasize visual fidelity or static 3D reconstruction, lacking systematic assessment of spatiotemporal interactive responsiveness. This work proposes the first comprehensive evaluation framework tailored to the 4D generation paradigm, introducing Omni-WorldSuite—a prompt suite encompassing multi-level interactions and diverse scene types—and Omni-Metrics, an agent-based causal influence measurement system that analyzes spatiotemporal state trajectories. Through large-scale evaluation of 18 leading models, the study reveals significant deficiencies in current approaches' ability to model interactive responses. The released Omni-WorldBench aims to catalyze progress in interactive world modeling by providing a standardized, holistic benchmark for future research.

This work addresses the challenge of safety evaluation for autonomous driving in long-tail scenarios, where existing neural simulators exhibit limited generalization. The authors propose a real-time generative world model based on an action-conditioned autoregressive diffusion framework, leveraging the large-scale video diffusion model Cosmos—adapted via mid-to-post training for autonomous driving simulation. Trained on 21,000 hours of driving data, the model enables realistic synthesis of unseen scenarios, including extreme weather and unpredictable dynamic behaviors. Integrated into a closed-loop system with the Alpamayo 1 policy and the AlpaSim coordinator, the approach significantly outperforms a vision-language-action (VLA) policy model with five times more parameters on the NuRec benchmark, demonstrating its strong potential as a policy backbone.

This work proposes a zero-dependency, cross-platform solution for real-time 3D scientific visualization that addresses the limitations of traditional tools—such as poor cross-platform support, high development barriers, and substantial maintenance costs—by leveraging the web browser as a unified rendering backend. The approach reuses existing OpenGL code via WebAssembly and integrates a built-in HTTP server to enable both local and remote interaction. Requiring no external dependencies, the framework seamlessly supports Jupyter Notebook, standalone browser environments, and conventional OpenGL contexts. Demonstrated through its integration with REBOUND, the solution achieves full platform compatibility, low usability barriers, and high maintainability, offering significant value for scientific research, education, and remote collaboration.

This work addresses the inability of CUDA and Vulkan to execute compute and graphics tasks concurrently on GPUs due to scheduling isolation, which severely limits hardware utilization. To overcome this limitation, the authors present the first cross-ecosystem spatial sharing solution between CUDA and Vulkan. Their approach introduces driver-level mechanisms—including channel redirection, virtual address space merging, and page table grafting—to unify scheduling and memory address spaces without requiring data copies. A lightweight developer annotation API is also provided to facilitate integration. Evaluated on representative embodied AI workloads, the system achieves up to 85% higher throughput compared to a time-multiplexing baseline, while significantly reducing end-to-end latency and improving overall GPU utilization.