PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving
Source abstract
Abstract
Owing to its wide range of applications, 3D object detection has attracted increasing attention in computer vision tasks. Most existing 3D object detection methods are based on Lidar point cloud data. However, these methods have some limitations in localization consistency and classification confidence, due to the irregularity and sparsity of Light Detection and Ranging (LiDAR) point cloud data. Inspired by the complementary characteristics of Lidar and camera sensors, we propose a new end-to-end learnable framework named Point-Image Fusion Network (PIFNet) to integrate the LiDAR point cloud and camera images. To resolve the problem of inconsistency in the localization and classification, we designed an Encoder-Decoder Fusion (EDF) module to extract the image features effectively, while maintaining the fine-grained localization information of objects. Furthermore, a new effective fusion module is proposed to integrate the color and texture features from images and the depth information from the point cloud. This module can enhance the irregularity and sparsity problem of the point cloud features by capitalizing the fine-grained information from camera images. In PIFNet, each intermediate feature map is fed into the fusion module to be integrated with its corresponding point-wise features. Furthermore, point-wise features are used instead of voxel-wise features to reduce information loss. Extensive experiments using the KITTI dataset demonstrate the superiority of PIFNet over other state-of-the-art methods. Compared with several state-of-the-art methods, our approach outperformed by 1.97% in mean Average Precision (mAP) and by 2.86% in Average Precision (AP) for the hard cases on the KITTI 3D object detection benchmark.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Robotics and Sensor-Based Localization · Remote Sensing and LiDAR Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering, Hanyang University, Ansan 15588, Korea
School of Electrical and Electronic Engineering, Hubei University of Technology, Wuhan 430068, China
Funding
Ministry of Trade, Industry and Energy
References (34)
Show source-provided references
1. Shuran, S., and Jianxiong, X. (2014, January 5–12). Sliding Shapes for 3D object detection in depth images. Proceedings of the European Conference on Computer Vision—ECCV, Zurich, Switzerland.
2. Dominic, Z.W., and Ingmar, P. (2015, January 13–17). Voting for voting in online point cloud object detection. Proceedings of the Robotics: Science and Systems XI, Rome, Italy.
3. A dynamic clustering algorithm for Lidar obstacle detection of autonomous driving system10.1109/JSEN.2021.3118365 ↗
4. Shuran, S., and Jianxiong, X. (July, January 26). Deep sliding shapes for amodal 3D object detection in RGB-D. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
5. Yang, Z., Sun, Y., Liu, S., Shen, X., and Jia, J. (2018). Ipod: Intensive point-based object detector for point cloud. arXiv.10.1109/ICCV.2019.00204 ↗
6. Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., and Li, H. (2020, January 13–19). PV-RCNN: Point-voxel feature set abstraction for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01054 ↗
7. Shi, S., Jiang, L., Deng, J., Wang, Z., Guo, C., Shi, J., Wang, X., and Li, H. (2021). PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection. arXiv.10.1109/CVPR42600.2020.01054 ↗
8. Jiajun, D., Shaoshuai, S., Peiwei, L., Wengang, Z., Yanyong, Z., and Houqiang, L. (2020). Voxel R-CNN: Towards high performance voxel-based 3d object detection. arXiv.
9. Wang, Z., and Jia, K. (2019, January 3–8). Frustum convnet: Sliding frustums to aggregate local point-wise features for amodal 3d object detection. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8968513 ↗
10. Qi, C.R., Hao, S., Kaichun, M., Mo, K., and Guibas, L.J. (2017, January 21–26). Pointnet: Deep learning on point sets for 3D classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.
11. Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017). Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv.
12. Runzhou, G., Zhuangzhuang, D., Yihan, H., Yu, W., Sijia, C., Li, H., and Yuan, L. (2020). Afdet: Anchor free one stage 3d object detection. arXiv.
13. Chenhang, H., Hui, Z., Jianqiang, H., Xian-Sheng, H., and Lei, Z. (2020, January 13–19). Structure aware single-stage 3d object detection from point cloud. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01189 ↗
14. Shi, S., Wang, X., and Li, H. (2019, January 15–20). Pointrcnn: 3d object proposal generation and detection from point cloud. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00086 ↗
15. Yang, Z., Sun, Y., Liu, S., Shen, X., and Jia, J. (November, January 27). Std: Sparse-to-dense 3d object detector for point cloud. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
16. Yan, Y., Mao, Y., and Li, B. (2018). Second: Sparsely embedded convolutional detection. Sensors, 18.10.3390/s18103337 ↗
17. Wang, Y., Fathi, A., Kundu, A., Ross, D.A., Pantofaru, C., Funkhouser, T., and Solomon, J. (2020, January 23–28). Pillar-based object detection for autonomous driving. Proceedings of the 16th European Conference of Computer Vision–ECCV, Glasgow, UK.10.1007/978-3-030-58542-6_2 ↗
18. Shi, S., Wang, Z., Wang, X., and Li, H. (2019). Part-A2 net: 3d part-aware and aggregation neural network for object detection from point cloud. arXiv.
19. Barrera, A., Guindel, C., Beltrán, J., and García, F. (2020, January 20–23). BirdNet+: End-to-end 3D object detection in LiDAR Bird’s Eye View. Proceedings of the IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.10.1109/ITSC45102.2020.9294293 ↗
20. Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., and Urtasun, R. (July, January 26). Monocular 3d object detection for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
21. Bin, X., and Zhenzhong, C. (2018, January 18–22). Multi-level fusion based 3d object detection from monocular images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.
22. 3d object proposals using stereo imagery for accurate object class detection10.1109/TPAMI.2017.2706685 ↗
23. Peiliang, L., Xiaozhi, C., and Shaojie, S. (2019, January 15–20). Stereo R-CNN based 3D Object Detection for Autonomous Driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00783 ↗
24. Liang, M., Yang, B., Chen, Y., Hu, R., and Urtasun, R. (2019, January 15–20). Multi-task multi-sensor fusion for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00752 ↗
25. Liang, M., Yang, B., Wang, S., and Urtasun, R. (2018, January 8–14). Deep continuous fusion for multi-sensor 3d object detection. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01270-0_39 ↗
26. Ku, J., Mozifian, M., Lee, J., Harakeh, A., and Waslander, S.L. (2018, January 1–5). Joint 3d proposal generation and object detection from view aggregation. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594049 ↗
27. Huang, T., Liu, Z., Chen, X., and Bai, X. (2020, January 23–28). Epnet: Enhancing point features with image semantics for 3D object detection. Proceedings of the European Conference on Computer Vision—ECCV, Glasgow, UK.10.1007/978-3-030-58555-6_3 ↗
28. Pang, S., Morris, D., and Radha, H. (January, January 24). CLOCs: Camera-LiDAR object candidates fusion for 3D object detection. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.
29. Yoo, J.H., Kim, Y., Kim, J., and Choi, J.W. (2020, January 23–28). 3D-CVF: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. Proceedings of the 16th European Conference of Computer Vision–ECCV, Glasgow, UK.10.1007/978-3-030-58583-9_43 ↗
30. Charles, R.Q., Wei, L., Chenxia, W., Hao, S., and Leonidas, J.G. (2018, January 18–22). Frustum PointNets for 3D Object Detection from RGB-D Data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.
31. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-view 3d object detection network for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
32. Meyer, G.P., Charland, J., Hegde, D., Laddha, A., and Vallespi-Gonzalez, C. (2019, January 15–20). Sensor fusion for joint 3D object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPRW.2019.00162 ↗
33. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the International Conference on Machine Learning, Lille, France.
34. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗