Unposed-to-3D: Learning Simulation-Ready Vehicles from Real-World Images

2026-04-21

AI Summary

Existing methods for generating 3D vehicle models rely on synthetic data, suffering from domain shift, arbitrary poses, and undefined scale, which lead to poor visual consistency in real-world driving scenarios. This work proposes a two-stage self-supervised framework that first trains a reconstruction network using pose-annotated images and then enables 3D reconstruction from unposed real-world images by predicting camera parameters. The approach incorporates scale-awareness and appearance harmonization modules to ensure geometric and photometric fidelity. To the best of our knowledge, this is the first method to generate simulation-ready 3D vehicle assets—exhibiting consistent poses, physically plausible scale, and lighting-coherent appearance—using only unposed real driving images in an end-to-end manner, thereby significantly enhancing the quality and scalability of autonomous driving simulation assets.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Web data generation and simulation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Creating realistic and simulation-ready 3D assets is crucial for autonomous driving research and virtual environment construction. However, existing 3D vehicle generation methods are often trained on synthetic data with significant domain gaps from real-world distributions. The generated models often exhibit arbitrary poses and undefined scales, resulting in poor visual consistency when integrated into driving scenes. In this paper, we present Unposed-to-3D, a novel framework that learns to reconstruct 3D vehicles from real-world driving images using image-only supervision. Our approach consists of two stages. In the first stage, we train an image-to-3D reconstruction network using posed images with known camera parameters. In the second stage, we remove camera supervision and use a camera prediction head that directly estimates the camera parameters from unposed images. The predicted pose is then used for differentiable rendering to provide self-supervised photometric feedback, enabling the model to learn 3D geometry purely from unposed images. To ensure simulation readiness, we further introduce a scale-aware module to predict real-world size information, and a harmonization module that adapts the generated vehicles to the target driving scene with consistent lighting and appearance. Extensive experiments demonstrate that Unposed-to-3D effectively reconstructs realistic, pose-consistent, and harmonized 3D vehicle models from real-world images, providing a scalable path toward creating high-quality assets for driving scene simulation and digital twin environments.

Problem

• 3D vehicle reconstruction
• real-world images
• simulation-ready assets
• pose consistency
• domain gap

Innovation

• unposed image
• 3D vehicle reconstruction
• simulation-ready assets
• self-supervised learning
• differentiable rendering