B4DL: A Benchmark for 4D LiDAR LLM in Spatio-Temporal Understanding

2025-08-07

AI Summary

To address the scarcity of high-quality annotations and the lack of multimodal large language model (MLLM) architectures tailored for 4D LiDAR point clouds in dynamic outdoor spatiotemporal understanding, this paper introduces B4DL—the first multimodal benchmark for 4D LiDAR—alongside an end-to-end 4D point cloud–language alignment framework. Methodologically, we design an MLLM architecture that natively accepts raw 4D LiDAR sequences, incorporate a spatiotemporal feature alignment mechanism, and develop an automated pipeline for synthetic data generation and annotation. Our contributions are threefold: (1) an open-source repository featuring rendered videos, a synthetic 4D LiDAR dataset, and multi-scenario reasoning outputs; (2) effective 4D geometric-semantic joint modeling; and (3) state-of-the-art performance on dynamic object interaction and scene evolution reasoning, significantly outperforming existing approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Understanding dynamic outdoor environments requires capturing complex object interactions and their evolution over time. LiDAR-based 4D point clouds provide precise spatial geometry and rich temporal cues, making them ideal for representing real-world scenes. However, despite their potential, 4D LiDAR remains underexplored in the context of Multimodal Large Language Models (MLLMs) due to the absence of high-quality, modality-specific annotations and the lack of MLLM architectures capable of processing its high-dimensional composition. To address these challenges, we introduce B4DL, a new benchmark specifically designed for training and evaluating MLLMs on 4D LiDAR understanding. In addition, we propose a scalable data generation pipeline and an MLLM model that, for the first time, directly processes raw 4D LiDAR by bridging it with language understanding. Combined with our dataset and benchmark, our model offers a unified solution for spatio-temporal reasoning in dynamic outdoor environments. We provide rendered 4D LiDAR videos, generated dataset, and inference outputs on diverse scenarios at: https://mmb4dl.github.io/mmb4dl/

Problem

• Lack of high-quality 4D LiDAR annotations for MLLMs
• No MLLM architectures for high-dimensional 4D LiDAR processing
• Need for spatio-temporal reasoning in dynamic outdoor environments

Innovation

• Introduces B4DL benchmark for 4D LiDAR MLLMs
• Proposes scalable 4D LiDAR data generation pipeline
• Develops MLLM model processing raw 4D LiDAR