MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving

2026-06-23

AI Summary

Existing methods struggle to fuse multi-view images and LiDAR point clouds for generating geometrically accurate and high-fidelity 3D vehicle models in real-world driving scenarios. This work proposes MM-TRELLIS, the first approach to incorporate LiDAR point clouds as test-time guidance within a native 3D diffusion generative model. By conditioning on multi-view images and enforcing geometric alignment during the denoising process, MM-TRELLIS achieves highly consistent generation through multimodal fusion. Additionally, it introduces a voxel filtering strategy based on 3D Gaussian Splatting opacities to effectively suppress floating artifacts. Evaluated on the Waymo dataset, the method significantly outperforms existing approaches, achieving state-of-the-art performance in fidelity, geometric accuracy, and cross-view consistency of the generated vehicle models.

Abstract

Recovering realistic 3D vehicle models from autonomous driving scenes is crucial for synthesizing training data and building simulation environment. However, most existing vehicle generation methods fail to fully exploit multimodal sensors i.e. multi-view images and LiDAR point clouds) and rely on neural rendering based reconstruction, leading to low-quality mesh. Recently, native 3D generative models have made significant progress, yet they are not built for arbitrary multi-view inputs and often struggle with in-the-wild driving images. In this work, we present MM-TRELLIS, a multi-modal version of TRELLIS for in-the-wild 3D vehicle generation that integrates LiDAR and image sensors from autonomous driving datasets into native 3D generative models. Specifically, multi-view images are cycled as conditioning inputs, while LiDAR point clouds provide test-time guidance to ensure geometric accuracy and cross-view consistency. During denoising, we first align the guidance point cloud with the model priors, then enforce consistency between the generated geometry and the guidance point cloud. Finally, we introduce a voxel filtering strategy based on the opacity of 3D Gaussian Splatting to suppress floaters and produce clean meshes. Comprehensive experiments on Waymo dataset demonstrate our method outperforms existing methods in high-fidelity 3D vehicle generation. Code is available at https://github.com/HongliXiao/MM-TRELLIS.

Problem

• 3D vehicle generation
• multi-modal sensing
• LiDAR point clouds
• autonomous driving
• mesh quality

Innovation

• multi-modal generation
• LiDAR guidance
• 3D Gaussian Splatting
• native 3D generative model
• voxel filtering