SEED4D: A Synthetic Ego-Exo Dynamic 4D Data Generator, Driving Dataset and Benchmark

2024-12-01

arXiv.org

AI Summary

Existing 3D/4D reconstruction methods for autonomous driving are severely limited by the scarcity of synchronized, multi-view, multi-temporal ego/exo RGB and LiDAR data in real-world settings. Method: This paper introduces the first synthetic dynamic multi-view 4D dataset generator and benchmark tailored for autonomous driving, featuring a customizable, open-source spatiotemporal multi-view synthesis framework. It supports physics-based sensor simulation, multi-camera calibration modeling, and configurations compatible with NuScenes, KITTI-360, and Waymo. Contribution/Results: We release 212K static images (2K scenes) and 16.8M dynamic images (10K trajectories × 100 frames), all with synchronized ego/exo RGB-LiDAR annotations. This benchmark fills a critical gap in multi-source temporal supervision for complex dynamic scenes, significantly improving generalization and cross-view consistency of few-shot 3D/4D reconstruction models, and advancing state-of-the-art performance across multiple benchmarks.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Web data generation and simulation
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Models for egocentric 3D and 4D reconstruction, including few-shot interpolation and extrapolation settings, can benefit from having images from exocentric viewpoints as supervision signals. No existing dataset provides the necessary mixture of complex, dynamic, and multi-view data. To facilitate the development of 3D and 4D reconstruction methods in the autonomous driving context, we propose a Synthetic Ego--Exo Dynamic 4D (SEED4D) data generator and dataset. We present a customizable, easy-to-use data generator for spatio-temporal multi-view data creation. Our open-source data generator allows the creation of synthetic data for camera setups commonly used in the NuScenes, KITTI360, and Waymo datasets. Additionally, SEED4D encompasses two large-scale multi-view synthetic urban scene datasets. Our static (3D) dataset encompasses 212k inward- and outward-facing vehicle images from 2k scenes, while our dynamic (4D) dataset contains 16.8M images from 10k trajectories, each sampled at 100 points in time with egocentric images, exocentric images, and LiDAR data. The datasets and the data generator can be found at https://seed4d.github.io/.

Problem

• 3D and 4D model reconstruction
• autonomous driving technology
• image acquisition from multiple angles and time points

Innovation

• SEED4D
• Multi-perspective Imaging
• Autonomous Driving Data