VirtualFilter: A High-Performance Multimodal 3D Object Detection Method with Semantic Filtering
Source abstract
Abstract
Three-dimensional object detection is a key task in the field of autonomous driving that is aimed at identifying the position and category of objects in the scene. Due to the 3D nature of data generated by LiDAR, most models use it as input data for detection. However, the low scanning resolution of LiDAR for distant objects has inherent limitations to the method, and multimodal fusion 3D object detection methods have attracted widespread attention, mostly using both LiDAR and camera data as inputs for detection. Certainly, multimodal methods can also lead to many problems, the two main ones being the incomplete utilization of camera features and rough fusion methods. In this study, we proposed a novel multimodal 3D object detection method named VirtualFilter, which uses 3D point clouds and 2D images as inputs. In order to better utilize camera features, VirtualFilter utilizes the image semantic segmentation model to generate image semantic features and uses the semantic information to filter the virtual point cloud data during the virtual point cloud generation process to enhance the data accuracy of the virtual cloud. In addition, VirtualFilter utilizes a better RoI feature fusion strategy named 3D-DGAF (3D Distance-based Grid Attentional Fusion), which employs a attention mechanism based on distance gridding to better fuse the RoI features of the original and virtual point clouds. The experimental results on the authoritative autonomous driving dataset KITTI show that this multimodal 3D object detection method outperforms the baseline method in several evaluation metrics.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Video Surveillance and Tracking Methods · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Computing, Harbin Institute of Technology, Harbin 150040, China
Chongqing Research Institute of HIT, Chongqing 400000, China
Funding
National Key Research and Development Program
References (29)
Show source-provided references
1. Multimodal virtual point 3d detection
2. Cai, D., Chen, K., Lin, Z., Li, D., Zhou, T., Ling, Y., and Leung, M.F. (2024). JointSTNet: Joint Pre-Training for Spatial-Temporal Traffic Forecasting. IEEE Trans. Consum. Electron.10.1109/TCE.2024.3476129 ↗
3. Pandey, A., Puri, M., and Varde, A. (2018, January 5–7). Object detection with neural models, deep learning and common sense to aid smart mobility. Proceedings of the 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), Volos, Greece.10.1109/ICTAI.2018.00134 ↗
4. Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (2019, January 15–20). Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01298 ↗
5. Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., and Li, H. (2020, January 13–19). PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection. Proceedings of the CVPR, Seattle, WA, USA.10.1109/CVPR42600.2020.01054 ↗
6. Wu, X., Peng, L., Yang, H., Xie, L., Huang, C., Deng, C., Liu, H., and Cai, D. (2022, January 18–24). Sparse fuse dense: Towards high quality 3d detection with depth completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00534 ↗
7. Wu, H., Wen, C., Shi, S., Li, X., and Wang, C. (2023, January 17–24). Virtual sparse convolution for multimodal 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02074 ↗
8. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-View 3D Object Detection Network for Autonomous Driving. Proceedings of the CVPR, Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
9. Vora, S., Lang, A.H., Helou, B., and Beijbom, O. (2020, January 13–19). Pointpainting: Sequential fusion for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00466 ↗
10. Beltrán, J., Guindel, C., Moreno, F.M., Cruzado, D., Garcia, F., and De La Escalera, A. (2018, January 4–7). Birdnet: A 3d object detection framework from lidar information. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.10.1109/ITSC.2018.8569311 ↗
11. Shi, S., Wang, X., and Li, H. (2019, January 15–20). Pointrcnn: 3d object proposal generation and detection from point cloud. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00086 ↗
12. Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., and Li, H. (2021, January 2–9). Voxel r-cnn: Towards high performance voxel-based 3d object detection. Proceedings of the AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v35i2.16207 ↗
13. Sindagi, V.A., Zhou, Y., and Tuzel, O. (2019, January 20–24). Mvx-net: Multimodal voxelnet for 3d object detection. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8794195 ↗
14. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
15. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany. proceedings, part III 18.
16. Segnet: A deep convolutional encoder-decoder architecture for image segmentation10.1109/TPAMI.2016.2644615 ↗
17. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
18. Fusion PSPnet image segmentation based method for multi-focus image fusion
19. SegFormer: Simple and efficient design for semantic segmentation with transformers
20. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
21. Hu, M., Wang, S., Li, B., Ning, S., Fan, L., and Gong, X. (June, January 30). Penet: Towards precise and efficient image guided depth completion. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561035 ↗
22. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
23. Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., and Qiao, Y. (2022). Vision transformer adapter for dense predictions. arXiv.
24. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
25. Zheng, W., Tang, W., Chen, S., Jiang, L., and Fu, C.W. (2021, January 2–9). Cia-ssd: Confident iou-aware single-stage object detector from point cloud. Proceedings of the AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v35i4.16470 ↗
26. He, C., Zeng, H., Huang, J., Hua, X.S., and Zhang, L. (2020, January 13–19). Structure Aware Single-stage 3D Object Detection from Point Cloud. Proceedings of the CVPR, Seattle, WA, USA.10.1109/CVPR42600.2020.01189 ↗
27. CasA: A cascade attention network for 3-D object detection from LiDAR point clouds10.1109/TGRS.2022.3228927 ↗
28. Yoo, J.H., Kim, Y., Kim, J., and Choi, J.W. (2020, January 23–28). 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. proceedings, part XXVII 16.10.1007/978-3-030-58583-9_43 ↗
29. Vpfnet: Improving 3d object detection with virtual point based lidar and stereo data fusion10.1109/TMM.2022.3189778 ↗