OmniRe: Omni Urban Scene Reconstruction

2024-08-29

arXiv.org

AI Summary

High-fidelity digital twin construction for diverse dynamic foreground objects—vehicles, pedestrians, and cyclists—in urban scenes remains challenging due to their heterogeneous motion patterns and appearance characteristics. Method: We propose the first unified 3D Gaussian Splatting (3DGS)-based scene graph framework supporting *all* dynamic foreground categories. Our approach establishes decoupled, object-specific Gaussian representations in canonical space, jointly optimized via multi-canonical-space modeling and end-to-end training, enabling real-time 60 Hz co-simulation. Contribution/Results: Unlike prior methods limited to vehicles, ours generalizes across dynamic agent types, significantly improving reconstruction completeness and physical consistency. It outperforms state-of-the-art on Waymo and demonstrates strong generalization and robustness across five major autonomous driving datasets—including complex urban scenarios. This advances human-vehicle interaction analysis and realistic human behavior simulation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: 3D Computer Vision

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce OmniRe, a comprehensive system for efficiently creating high-fidelity digital twins of dynamic real-world scenes from on-device logs. Recent methods using neural fields or Gaussian Splatting primarily focus on vehicles, hindering a holistic framework for all dynamic foregrounds demanded by downstream applications, e.g., the simulation of human behavior. OmniRe extends beyond vehicle modeling to enable accurate, full-length reconstruction of diverse dynamic objects in urban scenes. Our approach builds scene graphs on 3DGS and constructs multiple Gaussian representations in canonical spaces that model various dynamic actors, including vehicles, pedestrians, cyclists, and others. OmniRe allows holistically reconstructing any dynamic object in the scene, enabling advanced simulations (~60Hz) that include human-participated scenarios, such as pedestrian behavior simulation and human-vehicle interaction. This comprehensive simulation capability is unmatched by existing methods. Extensive evaluations on the Waymo dataset show that our approach outperforms prior state-of-the-art methods quantitatively and qualitatively by a large margin. We further extend our results to 5 additional popular driving datasets to demonstrate its generalizability on common urban scenes.

Problem

• Reconstruct diverse dynamic objects in urban scenes
• Enable advanced simulations with human-participated scenarios
• Outperform prior methods on fidelity and generalizability

Innovation

• Scene graphs on 3DGS for dynamic objects
• Multiple Gaussian representations in canonical spaces
• Holistic reconstruction enabling human-participated simulations