Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

2026-07-07

AI Summary

Existing autonomous driving simulation methods struggle to simultaneously achieve high visual fidelity and closed-loop interactivity, limiting the effective evaluation of end-to-end systems. To address this challenge, this work proposes a state-updating autoregressive generative framework that synthesizes driving videos conditioned on ego-vehicle states, traffic participants, scene maps, and point cloud skeletons. The approach introduces a Reset-and-Roll mechanism to mitigate error accumulation inherent in autoregressive inference. By leveraging point cloud skeletons to disentangle foreground and background content, and integrating rolling diffusion inference with a nuPlan-driven rendering-level closed-loop interface, the method significantly improves generation quality on the nuScenes and nuPlan datasets, enabling high-fidelity closed-loop simulation that combines photorealism with interactive responsiveness.

Abstract

Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes). We present \textbf{\emph{Point as Skeleton}}, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that \textit{Point as Skeleton} improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at https://github.com/krauwu/point-as-skeleton.

Problem

• autonomous driving simulation
• closed-loop simulation
• visual fidelity
• autoregressive generation
• point cloud

Innovation

• point-cloud skeleton
• autoregressive generation
• closed-loop simulation
• Reset-and-Roll
• driving video synthesis