Radar Voxel Fusion for 3D Object Detection
Source abstract
Abstract
Automotive traffic scenes are complex due to the variety of possible scenarios, objects, and weather conditions that need to be handled. In contrast to more constrained environments, such as automated underground trains, automotive perception systems cannot be tailored to a narrow field of specific tasks but must handle an ever-changing environment with unforeseen events. As currently no single sensor is able to reliably perceive all relevant activity in the surroundings, sensor data fusion is applied to perceive as much information as possible. Data fusion of different sensors and sensor modalities on a low abstraction level enables the compensation of sensor weaknesses and misdetections among the sensors before the information-rich sensor data are compressed and thereby information is lost after a sensor-individual object detection. This paper develops a low-level sensor fusion network for 3D object detection, which fuses lidar, camera, and radar data. The fusion network is trained and evaluated on the nuScenes data set. On the test set, fusion of radar data increases the resulting AP (Average Precision) detection score by about 5.1% in comparison to the baseline lidar network. The radar sensor fusion proves especially beneficial in inclement conditions such as rain and night scenes. Fusing additional camera data contributes positively only in conjunction with the radar fusion, which shows that interdependencies of the sensors are important for the detection result. Additionally, the paper proposes a novel loss to handle the discontinuity of a simple yaw representation for object detection. Our updated loss increases the detection and orientation estimation performance for all sensor input configurations. The code for this research has been made available on GitHub.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Optical Sensing Technologies · Robotics and Sensor-Based Localization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Automotive Technology, Technical University of Munich, 85748 Garching, Germany
mLab:Real-Time and Embedded Systems Lab, University of Pennsylvania, Philadelphia, PA 19104, USA
References (44)
Show source-provided references
1. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2019). nuScenes: A Multimodal Dataset for Autonomous Driving. arXiv.10.1109/CVPR42600.2020.01164 ↗
2. Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. (2013). Vision meets Robotics: The KITTI Dataset. Int. J. Robot. Res. IJRR.10.1177/0278364913491297 ↗
3. Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., and Li, H. (2019). PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection. arXiv.10.1109/CVPR42600.2020.01054 ↗
4. Kadar, I. (1997). New extension of the Kalman filter to nonlinear systems. Signal Processing, Sensor Fusion, and Target Recognition VI, SPIE. SPIE Proceedings.
5. Friedman, J.H. (2001). Greedy function approximation: A gradient boosting machine. Ann. Stat., 29.10.1214/aos/1013203451 ↗
6. Wang, L., Chen, T., Anklam, C., and Goldluecke, B. (November, January 19). High Dimensional Frustum PointNet for 3D Object Detection from Camera, LiDAR, and Radar. Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA.10.1109/IV47402.2020.9304655 ↗
7. Nobis, F., Fent, F., Betz, J., and Lienkamp, M. (2021). Kernel Point Convolution LSTM Networks for Radar Point Cloud Segmentation. Appl. Sci., 11.10.3390/app11062599 ↗
8. Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21–26). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
9. Qi, C.R., Liu, W., Wu, C., Su, H., and Guibas, L.J. (2018, January 18–23). Frustum PointNets for 3D Object Detection from RGB-D Data. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00102 ↗
10. Zhou, Y., and Tuzel, O. (2018, January 18–23). VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00472 ↗
11. Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., and Garnett, R. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28, Curran Associates, Inc.
12. Graham, B., and van der Maaten, L. (2017). Submanifold Sparse Convolutional Networks. arXiv.10.1109/CVPR.2018.00961 ↗
13. Graham, B., Engelcke, M., and van der Maaten, L. (2017). 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. arXiv.10.1109/CVPR.2018.00961 ↗
14. Yin, T., Zhou, X., and Krähenbühl, P. (2020). Center-based 3D Object Detection and Tracking. arXiv.10.1109/CVPR46437.2021.01161 ↗
15. Chadwick, S., Maddern, W., and Newman, P. (2019). Distant Vehicle Detection Using Radar and Vision. arXiv.10.1109/ICRA.2019.8794312 ↗
16. Nobis, F., Geisslinger, M., Weber, M., Betz, J., and Lienkamp, M. (2019, January 15–17). A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection. Proceedings of the 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany.10.1109/SDF.2019.8916629 ↗
17. Kowol, K., Rottmann, M., Bracke, S., and Gottschalk, H. (2020). YOdar: Uncertainty-Based Sensor Fusion for Vehicle Detection with Camera and Radar Sensors. arXiv.10.5220/0010239301770186 ↗
18. Kim, J., Kim, Y., and Kum, D. (December, January 30). Low-level Sensor Fusion Network for 3D Vehicle Detection using Radar Range-Azimuth Heatmap and Monocular Image. Proceedings of the Asian Conference on Computer Vision (ACCV) 2020, Kyoto, Japan.10.1007/978-3-030-69535-4_24 ↗
19. Nabati, R., and Qi, H. (2020). CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection. arXiv.10.1109/WACV48630.2021.00157 ↗
20. Lim, T.Y., Ansari, A., Major, B., Daniel, F., Hamilton, M., Gowaikar, R., and Subramanian, S. (2019, January 8–14). Radar and Camera Early Fusion for Vehicle Detection in Advanced Driver Assistance Systems. Proceedings of the Machine Learning for Autonomous Driving Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
21. Kim, Y., Choi, J.W., and Kum, D. (January, January 24). GRIF Net: Gated Region of Interest Fusion Network for Robust 3D Object Detection from Radar Point Cloud and Monocular Image. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341177 ↗
22. Ku, J., Mozifian, M., Lee, J., Harakeh, A., and Waslander, S. (2018, January 1–5). Joint 3D Proposal Generation and Object Detection from View Aggregation. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594049 ↗
23. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-View 3D Object Detection Network for Autonomous Driving. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
24. Vora, S., Lang, A.H., Helou, B., and Beijbom, O. (2020, January 13–19). PointPainting: Sequential Fusion for 3D Object Detection. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00466 ↗
25. Yang, B., Guo, R., Liang, M., Casas, S., and Urtasun, R. (2020). RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects. arXiv.10.1007/978-3-030-58523-5_29 ↗
26. Daniel, L., Phippen, D., Hoare, E., Stove, A., Cherniakov, M., and Gashinova, M. (2017, January 23–26). Low-THz Radar, Lidar and Optical Imaging through Artificially Generated Fog. Proceedings of the International Conference on Radar Systems (Radar 2017), Belfast, Ireland.10.1049/cp.2017.0369 ↗
27. Kamerman, G., and Steinvall, O. (2018, January 12–13). Mitigation of crosstalk effects in multi-LiDAR configurations. Proceedings of the Electro-Optical Remote Sensing XII, Berlin, Germany.
28. Kim, G., Eom, J., and Park, Y. (July, January 28). Investigation on the occurrence of mutual interference between pulsed terrestrial LIDAR scanners. Proceedings of the 2015 IEEE Intelligent Vehicles Symposium (IV), Seoul, Korea.10.1109/IVS.2015.7225724 ↗
29. Ng, A. (2021, June 16). A Chat with Andrew on MLOps: From Model-Centric to Data-Centric AI. Available online: https://www.youtube.com/watch?v=06-AZXmwHjo.
30. Zhou, Y., Barnes, C., Lu, J., Yang, J., and Li, H. (2018). On the Continuity of Rotation Representations in Neural Networks. arXiv.10.1109/CVPR.2019.00589 ↗
31. Yan, Y., Mao, Y., and Li, B. (2018). SECOND: Sparsely Embedded Convolutional Detection. Sensors, 18.10.3390/s18103337 ↗
32. Shorten, C., and Khoshgoftaar, T.M. (2019). A survey on Image Data Augmentation for Deep Learning. J. Big Data, 6.10.1186/s40537-019-0197-0 ↗
33. Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks10.1038/s41598-019-52737-x ↗
34. You, Y., Wang, Y., Chao, W.L., Garg, D., Pleiss, G., Hariharan, B., Campbell, M., and Weinberger, K.Q. (2019). Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving. arXiv.
35. Ku, J., Harakeh, A., and Waslander, S.L. (2018, January 8–10). In Defense of Classical Image Processing: Fast Depth Completion on the CPU. Proceedings of the 2018 15th Conference on Computer and Robot Vision (CRV), Toronto, ON, Canada.10.1109/CRV.2018.00013 ↗
36. A Research of Stereo Vision Positioning under Vibration
37. Fan, R., Wang, L., Bocus, M.J., and Pitas, I. (2020). Computer Stereo Vision for Autonomous Driving. arXiv.
38. Instruments, T. (2021, June 16). Stereo Vision-Facing the Challenges and Seeing the Opportunities for ADAS (Rev. A). Available online: https://www.ti.com/lit/wp/spry300a/spry300a.pdf?ts=1623899849893&ref_url=https%253A%252F%252Fwww.google.com%252F.
39. Wang, Y., Chao, W.L., Garg, D., Hariharan, B., Campbell, M., and Weinberger, K.Q. (2018). Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. arXiv.10.1109/CVPR.2019.00864 ↗
40. Nobis, F., Brunhuber, F., Janssen, S., Betz, J., and Lienkamp, M. (2020, January 20–23). Exploring the Capabilities and Limits of 3D Monocular Object Detection—A Study on Simulation and Real World Data. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.10.1109/ITSC45102.2020.9294625 ↗
41. Monocular depth estimation based on deep learning: An overview10.1007/s11431-020-1582-8 ↗
42. Girshick, R. (2015, January 7–13). Fast R-CNN. Proceedings of the ICCV 2015, Santiago, Chile.10.1109/ICCV.2015.169 ↗
43. Scheiner, N., Schumann, O., Kraus, F., Appenrodt, N., Dickmann, J., and Sick, B. (2020). Off-the-shelf sensor vs. experimental radar—How much resolution is necessary in automotive radar classification?. arXiv.10.23919/FUSION45008.2020.9190338 ↗
44. Shah, M., Huang, Z., Laddha, A., Langford, M., Barber, B., Zhang, S., Vallespi-Gonzalez, C., and Urtasun, R. (2020). LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion. arXiv.