Real-Time Object Detection and Distance Measurement Enhanced with Semantic 3D Depth Sensing Using Camera–LiDAR Fusion
Source abstract
Abstract
Camera and LiDAR data fusion has been a popular research area, especially in the field of autonomous vehicles. This study evaluates the efficiency and accuracy of different depth point extraction methods, including Point-by-Point (PbyP), Complete Region Depth Extraction (CoRDE), Central Region Depth Extraction (CeRDE), and Grid Central Region Depth Extraction (GCRDE), across object categories such as person, bicycle, car, bus, and truck, and occlusion levels ranging from 0 to 3. The approaches are assessed based on extraction time, accuracy, and root mean squared error (RMSE). Bounding box-based methods, such as PbyP and CoRDE, consistently show slower extraction times compared to segmentation mask methods, with CeRDE being the most efficient in terms of computational speed. However, segmentation mask methods, particularly CeRDE and GCRDE, offer superior accuracy, especially for complex objects like trucks and cars, where bounding box methods struggle, particularly at higher occlusion levels. In terms of RMSE, segmentation mask methods consistently outperform bounding box methods, providing more precise depth estimations, particularly for larger and more occluded objects. Overall, segmentation mask methods are preferred for applications where accuracy is critical, despite their slower processing speed, while bounding box methods are suitable for real-time applications requiring faster depth extraction. GeRDE offers a balance between speed and accuracy, making it ideal for tasks needing both efficiency and precision.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Robotics and Sensor-Based Localization · Advanced Optical Sensing Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronic and Electrical Engineering, Brunel University of London, London UB8 3PH, UK
Funding
Ministry of National Education of Türkiye
References (32)
Show source-provided references
1. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The kitti vision benchmark suite. Proceedings of the 2012 IEEE conference on computer vision and pattern recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
2. Qi, C.R., Liu, W., Wu, C., Su, H., and Guibas, L.J. (2018, January 18–23). Frustum pointnets for 3d object detection from rgb-d data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00102 ↗
3. Ku, J., Pon, A.D., and Waslander, S.L. (2019, January 15–20). Monocular 3d object detection leveraging accurate proposals and shape reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01214 ↗
4. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-view 3d object detection network for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
5. Vora, S., Lang, A.H., Helou, B., and Beijbom, O. (2020, January 13–19). Pointpainting: Sequential fusion for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00466 ↗
6. Nabati, R., and Qi, H. (2020). Radar-camera sensor fusion for joint object detection and distance estimation in autonomous vehicles. arXiv.
7. Favelli, S., Xie, M., and Tonoli, A. (2024). Sensor Fusion Method for Object Detection and Distance Estimation in Assisted Driving Applications. Sensors, 24.10.3390/s24247895 ↗
8. Ma, F., Cavalheiro, G.V., and Karaman, S. (2019, January 20–24). Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793637 ↗
9. Mai, N.A.M., Duthon, P., Khoudour, L., Crouzil, A., and Velastin, S.A. (2021, January 22–23). Sparse LiDAR and stereo fusion (SLS-Fusion) for depth estimation and 3D object detection. Proceedings of the IET Conference Proceedings CP773. IET, Tianjin, China.10.1049/icp.2021.1442 ↗
10. Liu, M., Jia, Y., Lyu, Y., Dong, Q., and Yang, Y. (2024). BAFusion: Bidirectional Attention Fusion for 3D Object Detection Based on LiDAR and Camera. Sensors, 24.10.3390/s24144718 ↗
11. Xiang, X., and Zhang, J. (2023). Fusionvit: Hierarchical 3d object detection via lidar-camera vision transformer fusion. arXiv.
12. CLFT: Camera-LiDAR Fusion Transformer for Semantic Segmentation in Autonomous Driving10.1109/TIV.2024.3454971 ↗
13. Lcpr: A multi-scale attention-based lidar-camera fusion network for place recognition10.1109/LRA.2023.3346753 ↗
14. Xiao, Y., Li, Y., Meng, C., Li, X., Ji, J., and Zhang, Y. (2024, January 13–17). Calibformer: A transformer-based automatic lidar-camera calibration network. Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan.10.1109/ICRA57147.2024.10610018 ↗
15. Dong, H., Gu, W., Zhang, X., Xu, J., Ai, R., Lu, H., Kannala, J., and Chen, X. (2024, January 13–17). Superfusion: Multilevel lidar-camera fusion for long-range hd map generation. Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan.10.1109/ICRA57147.2024.10611320 ↗
16. Dam, T., Dharavath, S.B., Alam, S., Lilith, N., Chakraborty, S., and Feroskhan, M. (2024, January 13–17). AYDIV: Adaptable Yielding 3D Object Detection via Integrated Contextual Vision Transformer. Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan.10.1109/ICRA57147.2024.10610908 ↗
17. Kumar, G.A., Lee, J.H., Hwang, J., Park, J., Youn, S.H., and Kwon, S. (2020). LiDAR and camera fusion approach for object distance estimation in self-driving vehicles. Symmetry, 12.10.3390/sym12020324 ↗
18. Wang, C.Y., Bochkovskiy, A., and Liao, H.Y.M. (2023, January 17–24). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00721 ↗
19. Varghese, R., and Sambath, M. (2024, January 18–19). Yolov8: A novel object detection algorithm with enhanced performance and robustness. Proceedings of the 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), Chennai, India.10.1109/ADICS58448.2024.10533619 ↗
20. Wang, C.Y., Yeh, I.H., and Mark Liao, H.Y. (2024). Yolov9: Learning what you want to learn using programmable gradient information. European Conference on Computer Vision, Springer.10.1007/978-3-031-72751-1_1 ↗
21. Yolov10: Real-time end-to-end object detection
22. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
23. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
24. Zhang, Y., Guo, Z., Wu, J., Tian, Y., Tang, H., and Guo, X. (2022). Real-time vehicle detection based on improved yolo v5. Sustainability, 14.10.3390/su141912274 ↗
25. Li, C., Li, L., Jiang, H., Weng, K., Geng, Y., Li, L., Ke, Z., Li, Q., Cheng, M., and Nie, W. (2022). YOLOv6: A single-stage object detection framework for industrial applications. arXiv.
26. Vision meets Robotics: The KITTI Dataset10.1177/0278364913491297 ↗
27. Accuracy of airborne lidar-derived elevation10.14358/PERS.70.3.331 ↗
28. Lu, W., Zhou, Y., Wan, G., Hou, S., and Song, S. (2019, January 16–20). L3-Net: Towards Learning Based LiDAR Localization for Autonomous Driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00655 ↗
29. Taylor, Z., and Nieto, J. (2013, January 1–8). Automatic calibration of lidar and camera images using normalized mutual information. Proceedings of the Robotics and Automation (ICRA), 2013 IEEE International Conference on Citeseer, Sydney, Australia.
30. Wei, P., Cagle, L., Reza, T., Ball, J., and Gafford, J. (2018). LiDAR and camera detection fusion in a real-time industrial multi-sensor collision avoidance system. Electronics, 7.10.3390/electronics7060084 ↗
31. Probabilistic multi-modal depth estimation based on camera–LiDAR sensor fusion10.1007/s00138-023-01426-x ↗
32. Zhu, Z., Wu, F., Sun, W., Wu, Q., Liang, F., and Zhang, W. (2025). Depth Estimation Based on MMwave Radar and Camera Fusion with Attention Mechanisms and Multi-Scale Features for Autonomous Driving Vehicles. Electronics, 14.10.3390/electronics14020300 ↗