PRNet: 3D Object Detection Network-Based on Point-Region Fusion
Source abstract
Abstract
Object detection is a pivotal task in the realm of autonomous driving, where reliance on single-modality information often proves inadequate for high-precision detection tasks. In current research, object detection networks based on point clouds effectively identify objects with dense point clouds. However, these networks face challenges when dealing with detection tasks involving objects with higher point cloud sparsity. To address this issue, this paper proposes a 3D object detection network based on point cloud and image fusion. This network employs a fusion module named PRF (Point-Region Fusion), utilizing the K-Nearest Neighbors (KNN) algorithm to find the nearest K points corresponding to point cloud features. It then collects regional features from image feature maps and fuses them with the point cloud features after aggregation. The designed Image Feature Fusion module (IF-Fusion) fuses image feature maps of varying sizes in a pairwise manner. This fusion process is key in preserving features of small objects and subsequently enriching the point cloud’s features. In evaluations conducted on the KITTI benchmark, the method presented here surpasses prior fusion networks in detection accuracy, achieving rates of 91.95%, 81.10%, and 78.08%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Video Surveillance and Tracking Methods · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical and Electronic Engineering, University of Liverpool, Liverpool L3 5UB, UK
College of Information Engineering, East China Jiaotong University, Nanchang 330013, China
Funding
National Natural Science Foundation of China
References (43)
Show source-provided references
1. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? the kitti vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
2. Deep learning for lidar point clouds in autonomous driving: A review10.1109/TNNLS.2020.3015992 ↗
3. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 13–19). Scalability in perception for autonomous driving: Waymo open dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00252 ↗
4. Ataer-Cansizoglu, E., Taguchi, Y., Ramalingam, S., and Garaas, T. (2013, January 2). Tracking an RGB-D camera using points and planes. Proceedings of the IEEE International Conference on Computer Vision Workshops, Sydney, Australia.10.1109/ICCVW.2013.14 ↗
5. Spatiotemporal Texture Reconstruction for Dynamic Objects Using a Single RGB-D Camera10.1111/cgf.142652 ↗
6. Wang, Z., Zhan, W., and Tomizuka, M. (2018). Fusing bird’s eye view lidar point cloud and front view camera image for 3d object detection. 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China, 26–30 June 2018, IEEE.10.1109/IVS.2018.8500387 ↗
7. Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., and Urtasun, R. (2016, January 27–30). Monocular 3d object detection for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.236 ↗
8. 3d object proposals using stereo imagery for accurate object class detection10.1109/TPAMI.2017.2706685 ↗
9. Huang, T., Liu, Z., Chen, X., and Bai, X. (2020). Epnet: Enhancing point features with image semantics for 3d object detection. Computer Vision–ECCV 2020, 16th European Conference, Glasgow, UK, 23–28 August 2020, Proceedings, Part XV 16, Springer International Publishing.10.1007/978-3-030-58555-6_3 ↗
10. Liang, M., Yang, B., Wang, S., and Urtasun, R. (2018, January 8–14). Deep continuous fusion for multi-sensor 3d object detection. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01270-0_39 ↗
11. Survey and systematization of 3D object detection models and methods10.1007/s00371-023-02891-1 ↗
12. Mousavian, A., Anguelov, D., Flynn, J., and Kosecka, J. (2017, January 21–26). 3d bounding box estimation using deep learning and geometry. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.597 ↗
13. Li, B., Ouyang, W., Sheng, L., Zeng, X., and Wang, X. (2019, January 15–20). Gs3d: An efficient 3d object detection framework for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00111 ↗
14. Shi, X., Ye, Q., Chen, X., Chen, C., Chen, Z., and Kim, T.K. (2021, January 11–17). Geometry-based distance decomposition for monocular 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01489 ↗
15. Monocular 3d object detection with decoupled structured polygon estimation and height-guided depth estimation
16. Ma, X., Zhang, Y., Xu, D., Zhou, D., Yi, S., Li, H., and Ouyang, W. (2021, January 20–25). Delving into localization errors for monocular 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00469 ↗
17. Learning auxiliary monocular contexts helps monocular 3D object detection10.1609/aaai.v36i2.20074 ↗
18. You, Y., Wang, Y., Chao, W.L., Garg, D., Pleiss, G., Hariharan, B., Campbell, M., and Weinberger, K.Q. (2019). Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. arXiv.
19. Qian, R., Garg, D., Wang, Y., You, Y., Belongie, S., Hariharan, B., Campbell, M., Weinberger, K.Q., and Chao, W.L. (2020, January 14–19). End-to-end pseudo-lidar for image-based 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00592 ↗
20. Guo, X., Shi, S., Wang, X., and Li, H. (2021, January 10–17). Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00314 ↗
21. Yang, Z., Sun, Y., Liu, S., and Jia, J. (2020, January 13–19). 3dssd: Point-based 3d single stage object detector. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01105 ↗
22. P2v-rcnn: Point to voxel feature learning for 3d object detection from point clouds10.1109/ACCESS.2021.3094562 ↗
23. Li, J., Luo, S., Zhu, Z., Dai, H., Krylov, A.S., Ding, Y., and Shao, L. (2020). 3D IoU-Net: IoU guided 3D object detector for point clouds. arXiv.
24. Sun, P., Wang, W., Chai, Y., Elsayed, G., Bewley, A., Zhang, X., Sminchisescu, C., and Anguelov, D. (2021, January 20–25). Rsn: Range sparse net for efficient, accurate lidar 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00567 ↗
25. Fan, L., Xiong, X., Wang, F., Wang, N., and Zhang, Z. (2021, January 11–17). Rangedet: In defense of range view for lidar-based 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00291 ↗
26. Liang, Z., Zhang, Z., Zhang, M., Zhao, X., and Pu, S. (2021, January 20–25). Rangeioudet: Range image based real-time 3d object detector optimized by intersection over union. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00706 ↗
27. Zhou, Y., and Tuzel, O. (2018, January 18–23). Voxelnet: End-to-end learning for point cloud based 3d object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00472 ↗
28. Wang, Y., Fathi, A., Kundu, A., Ross, D.A., Pantofaru, C., Funkhouser, T., and Solomon, J. (2020). Pillar-based object detection for autonomous driving. Computer Vision–ECCV 2020, 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, Springer International Publishing.10.1007/978-3-030-58542-6_2 ↗
29. Kuang, H., Wang, B., An, J., Zhang, M., and Zhang, Z. (2020). Voxel-FPN: Multi-scale voxel feature aggregation for 3D object detection from LIDAR point clouds. Sensors, 20.10.3390/s20030704 ↗
30. Voxel r-cnn: Towards high performance voxel-based 3d object detection
31. Zheng, W., Tang, W., Jiang, L., and Fu, C.W. (2021, January 20–25). SE-SSD: Self-ensembling single-stage object detector from point cloud. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01426 ↗
32. Behind the curtain: Learning occluded shapes for 3d object detection
33. BADet: Boundary-aware 3D object detection from point clouds10.1016/j.patcog.2022.108524 ↗
34. Shi, S., Wang, X., and Li, H. (2019, January 15–20). Pointrcnn: 3d object proposal generation and detection from point cloud. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00086 ↗
35. Li, B., Zhang, T., and Xia, T. (2016). Vehicle detection from 3d lidar using fully convolutional network. arXiv.10.15607/RSS.2016.XII.042 ↗
36. Yang, B., Liang, M., and Urtasun, R. (2018, January 29–31). Hdnet: Exploiting hd maps for 3d object detection. Proceedings of the Conference on Robot Learning, Zürich, Switzerland. PMLR.
37. Yang, B., Luo, W., and Urtasun, R. (2018, January 18–23). Pixor: Real-time 3d object detection from point clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00798 ↗
38. Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017). Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Adv. Neural Inf. Process. Syst., 30.
39. Qi, C.R., Liu, W., Wu, C., Su, H., and Guibas, L.J. (2018, January 18–23). Frustum pointnets for 3d object detection from rgb-d data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00102 ↗
40. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-view 3d object detection network for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
41. Ku, J., Mozifian, M., Lee, J., Harakeh, A., and Waslander, S.L. (2018, January 1–5). Joint 3d proposal generation and object detection from view aggregation. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594049 ↗
42. Sindagi, V.A., Zhou, Y., and Tuzel, O. (2019, January 20–24). Mvx-net: Multimodal voxelnet for 3d object detection. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8794195 ↗
43. CrossFusion net: Deep 3D object detection based on RGB images and point clouds in autonomous driving10.1016/j.imavis.2020.103955 ↗