Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data

2025-11-04

AI Summary

Virtual testing environments (VTEs) require high-fidelity simulation for reliable autonomous driving system (ADS) validation, necessitating rigorous quantification of geometric and perceptual fidelity between real and simulated LiDAR point clouds. Method: We propose the density-aware Chamfer distance (DCD), a novel metric that jointly accounts for geometric structure and perception-relevant density variations. DCD is evaluated under realistic perturbations—including noise, variable point density, and pose uncertainty—and benchmarked against standard metrics (e.g., Chamfer Distance, Earth Mover’s Distance). Real data are acquired via multi-sensor fusion (LiDAR/IMU/camera), enabling high-accuracy static VTE reconstruction and registration-consistent synthetic point cloud generation. DCD validity is further corroborated through point cloud similarity analysis and semantic segmentation (intensity-corrected mIoU = 21%). Results: Real–simulated scan pairs yield a mean DCD of 0.63—significantly lower than baseline metrics—and DCD exhibits the strongest correlation with downstream perception performance, establishing its reliability and novelty as a core VTE evaluation metric.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

For developing safe Autonomous Driving Systems (ADS), rigorous testing is required before they are deemed safe for road deployments. Since comprehensive conventional physical testing is impractical due to cost and safety concerns, Virtual Testing Environments (VTE) can be adopted as an alternative. Comparing VTE-generated sensor outputs against their real-world analogues can be a strong indication that the VTE accurately represents reality. Correspondingly, this work explores a comprehensive experimental approach to finding evaluation metrics suitable for comparing real-world and simulated LiDAR scans. The metrics were tested in terms of sensitivity and accuracy with different noise, density, distortion, sensor orientation, and channel settings. From comparing the metrics, we found that Density Aware Chamfer Distance (DCD) works best across all cases. In the second step of the research, a Virtual Testing Environment was generated using real LiDAR scan data. The data was collected in a controlled environment with only static objects using an instrumented vehicle equipped with LiDAR, IMU and cameras. Simulated LiDAR scans were generated from the VTEs using the same pose as real LiDAR scans. The simulated and LiDAR scans were compared in terms of model perception and geometric similarity. Actual and simulated LiDAR scans have a similar semantic segmentation output with a mIoU of 21% with corrected intensity and an average density aware chamfer distance (DCD) of 0.63. This indicates a slight difference in the geometric properties of simulated and real LiDAR scans and a significant difference between model outputs. During the comparison, density-aware chamfer distance was found to be the most correlated among the metrics with perception methods.

Problem

• Evaluating metrics for comparing real and simulated LiDAR data
• Assessing metric sensitivity to noise, density, and distortions
• Validating Virtual Testing Environments through LiDAR scan comparisons

Innovation

• Density Aware Chamfer Distance for LiDAR comparison
• Virtual Testing Environment with real sensor data
• Evaluating geometric and perception similarity metrics