Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception

2025-03-17

AI Summary

To address the disconnection between future scene generation and geometric perception in autonomous driving, this paper proposes UniFuture—a unified world model that jointly generates and perceives RGB images and depth maps for the first time. Methodologically, UniFuture introduces a dual latent space sharing mechanism and a multi-scale latent feature interaction module; it employs a variational autoencoder to establish bidirectional mapping between RGB and depth modalities, integrates cross-attention to enable mutual optimization of appearance and geometry, and adopts a joint reconstruction loss for end-to-end training. Evaluated on nuScenes, UniFuture significantly outperforms task-specific models: it reduces future video divergence (FVD) by 18.3% and depth estimation error by 12.7%. Crucially, it synthesizes spatiotemporally consistent future RGB-depth pairs from a single input frame—establishing a novel paradigm for holistic driving scene modeling.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present UniFuture, a simple yet effective driving world model that seamlessly integrates future scene generation and perception within a single framework. Unlike existing models focusing solely on pixel-level future prediction or geometric reasoning, our approach jointly models future appearance (i.e., RGB image) and geometry (i.e., depth), ensuring coherent predictions. Specifically, during the training, we first introduce a Dual-Latent Sharing scheme, which transfers image and depth sequence in a shared latent space, allowing both modalities to benefit from shared feature learning. Additionally, we propose a Multi-scale Latent Interaction mechanism, which facilitates bidirectional refinement between image and depth features at multiple spatial scales, effectively enhancing geometry consistency and perceptual alignment. During testing, our UniFuture can easily predict high-consistency future image-depth pairs by only using the current image as input. Extensive experiments on the nuScenes dataset demonstrate that UniFuture outperforms specialized models on future generation and perception tasks, highlighting the advantages of a unified, structurally-aware world model. The project page is at https://github.com/dk-liang/UniFuture.

Problem

• Unified model for future scene generation and perception.
• Joint modeling of future appearance and geometry for coherent predictions.
• Enhanced geometry consistency and perceptual alignment through multi-scale interaction.

Innovation

• Unified model for future scene generation and perception
• Dual-Latent Sharing scheme for shared feature learning
• Multi-scale Latent Interaction for geometry consistency