CARLA-GS: Decoupling Representation, Reasoning, and Physics Simulation for Autonomous Driving Corner-Case Synthesis

2026-07-08

AI Summary

This work addresses the critical need for generating extreme driving scenarios that simultaneously exhibit visual photorealism, semantic plausibility, and physical feasibility—qualities rarely achieved together by existing methods. The authors propose a modular synthesis framework that decouples high-level semantic control from low-level physical execution. Specifically, they construct editable 3D Gaussian scenes from real-world data, employ a multi-agent large language model to generate high-risk interactive intent trajectories, and execute these trajectories under dynamic constraints using a PID controller in CARLA. The resulting motions are back-projected into the Gaussian scene to render first-person-view videos. This framework is the first to unify semantic controllability, physical realism, and visual fidelity within a single system, demonstrating strong spatiotemporal consistency, semantic alignment, and photorealism on the Waymo Open Dataset.

Abstract

Safety evaluation for autonomous driving is dominated by rare, safety-critical interactions, motivating simulators that can deliberately synthesize corner cases with photorealistic observations. Corner-case generation is inherently a multi-source problem spanning visual representation, scene reasoning, and vehicle trajectory generation and control. Prior knowledge- and model-based approaches typically focus on scene or trajectory components in isolation, while diffusion-based methods attempt end-to-end generation but still struggle to ensure spatiotemporal consistency and physical realism. To unify these aspects within a single framework, we propose CARLA-GS, a modular corner-case synthesis pipeline that decouples visual representation, semantic reasoning, and physics-based execution while maintaining tight cross-module coupling. Starting from real driving data, we reconstruct an editable gaussian scene with additional geometry-consistent constraints. A multi-agent LLM then performs scene-level reasoning to identify risky interactions and generate intent-level waypoint trajectories, while the low-level motion control is delegated to CARLA, where a PID controller ensures kinematic and dynamic feasibility. The simulated vehicle states are finally re-projected into the gaussian scene for ego-centric rendering. This design enables high-level semantic reasoning, low-level physically executable motion, and photorealistic corner-case generation within a unified pipeline. Experiments on the Waymo Open Dataset show, both quantitatively and qualitatively, that our framework enables controllable corner-case generation and produces photorealistic, spatiotemporally consistent videos aligned with semantic intent and physically feasible motion.

Problem

• corner-case synthesis
• autonomous driving
• spatiotemporal consistency
• physical realism
• photorealistic simulation

Innovation

• corner-case synthesis
• gaussian scene
• modular decoupling
• semantic reasoning
• physics-based simulation