Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data
IEEE Open Journal of Intelligent Transportation Systems · 2026 · 7 · 1091-1106
DOI: 10.1109/ojits.2026.3673597 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Open Journal of Intelligent Transportation Systems, 2026. Verfasst von Syed Mostaquim Ali und 5 weiteren Autor:innen. Die Arbeit nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Motor Vehicle Safety Program of Transport Canada and the Resilient Ground Transportation Program of the National Research Council Canada.
Abstract (Originalsprache)
For developing safe Autonomous Driving Systems (ADS), rigorous testing is required before they are deemed safe for road deployments. Since comprehensive conventional physical testing is impractical due to cost and safety concerns, Virtual Testing Environments (VTE) can be adopted as an alternative. Comparing VTE-generated sensor outputs against their real-world analogues can be a strong indication that the VTE accurately represents reality. Correspondingly, this work explores a comprehensive experimental approach to finding evaluation metrics suitable for comparing real-world and simulated LiDAR scans. The metrics were tested in terms of sensitivity and accuracy with different noise, density, distortion, sensor orientation, and channel settings. From comparing the metrics, we found that Density Aware Chamfer Distance (DCD) works best across all cases. In the second step of the research, a Virtual Testing Environment was generated using real LiDAR scan data. The data was collected in a controlled environment with only static objects using an instrumented vehicle equipped with LiDAR, IMU and cameras. Simulated LiDAR scans were generated from the VTEs using the same pose as real LiDAR scans. The simulated and LiDAR scans were compared in terms of model perception and geometric similarity. Actual and simulated LiDAR scans have a similar semantic segmentation output with a mIoU of 21% with corrected intensity and an average density aware chamfer distance (DCD) of 0.63. This indicates a slight difference in the geometric properties of simulated and real LiDAR scans and a significant difference between model outputs. During the comparison, density-aware chamfer distance was found to be the most correlated among the metrics with perception methods.
Förderung
- Motor Vehicle Safety Program of Transport Canada and the Resilient Ground Transportation Program of the National Research Council Canada
Zitieren
APA
Ali, S. M., Rahman, T., Farhani, G., Zaki, M. H., Anctil, B., & Charlebois, D. (2026). Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data. IEEE Open Journal of Intelligent Transportation Systems, 7, 1091-1106. https://doi.org/10.1109/ojits.2026.3673597
Vancouver
Ali SM, Rahman T, Farhani G, Zaki MH, Anctil B, Charlebois D. Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data. IEEE Open Journal of Intelligent Transportation Systems. 2026;7:1091-1106. doi:10.1109/ojits.2026.3673597
BibTeX
@article{Ali2026,
author = {Ali, Syed Mostaquim and Rahman, Taufiq and Farhani, Ghazal and Zaki, Mohamed H. and Anctil, Benoit and Charlebois, Dominique},
title = {Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data},
journal = {IEEE Open Journal of Intelligent Transportation Systems},
year = {2026},
volume = {7},
pages = {1091--1106},
doi = {10.1109/ojits.2026.3673597}
}
EUVIMED