R-LiViT: A LiDAR-Visual-Thermal Dataset Enabling Vulnerable Road User Focused Roadside Perception

2025-03-21

AI Summary

Detecting vulnerable road users (VRUs) in roadside perception remains challenging under extreme lighting conditions due to occlusions and visual degradation. Method: We introduce the first roadside multimodal dataset specifically designed for VRU perception, concurrently capturing LiDAR, RGB, and thermal infrared data across three intersections under both daytime and nighttime conditions—encompassing over 150 real-world scenarios. We propose a novel pipeline for precise spatiotemporal alignment of all three modalities, including timestamp synchronization, spatial calibration, and frame-level cross-modal registration. Annotations include fine-grained VRU instance labels: six categories for LiDAR point clouds and eight for image modalities. Contribution/Results: The dataset comprises 10,000 annotated LiDAR frames and 2,400 aligned RGB–thermal image pairs, supporting detection and tracking tasks. All data and evaluation code are publicly released, addressing the absence of thermal imaging in roadside VRU perception and establishing a new benchmark for multimodal roadside intelligent sensing.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

In autonomous driving, the integration of roadside perception systems is essential for overcoming occlusion challenges and enhancing the safety of Vulnerable Road Users (VRUs). While LiDAR and visual (RGB) sensors are commonly used, thermal imaging remains underrepresented in datasets, despite its acknowledged advantages for VRU detection in extreme lighting conditions. In this paper, we present R-LiViT, the first dataset to combine LiDAR, RGB, and thermal imaging from a roadside perspective, with a strong focus on VRUs. R-LiViT captures three intersections during both day and night, ensuring a diverse dataset. It includes 10,000 LiDAR frames and 2,400 temporally and spatially aligned RGB and thermal images across over 150 traffic scenarios, with 6 and 8 annotated classes respectively, providing a comprehensive resource for tasks such as object detection and tracking. The dataset1 and the code for reproducing our evaluation results2 are made publicly available.

Problem

• Integrating thermal imaging for VRU detection in extreme lighting
• Combining LiDAR, RGB, thermal data from roadside perspective
• Addressing occlusion challenges in autonomous driving roadside perception

Innovation

• Combines LiDAR, RGB, and thermal imaging
• Focuses on VRU detection in varied lighting
• Includes 10,000 LiDAR and 2,400 aligned images