Temporal Correlation Optimization for Accelerated 3D Sparse Convolution in Point Cloud Inference on CPU
Source abstract
Abstract
3D point clouds primarily consist of irregular, sparse data, dominated by background elements. The inherent irregularity of 3D point clouds induces elevated data movement, while the predominance of background points significantly amplifies computational requirements. Inspired by the substantial overlap of background points in adjacent frames, we introduce a pruning technique that exploits temporal correlations across successive frames to reduce redundant computations and expedite inference. This methodology optimizes computational resources to process valuable and highly correlated data, rather than indiscriminately processing entire point clouds. To further accelerate performance, we enhance data movement using Single Instruction Multiple Data (SIMD) techniques, optimizing the time-intensive Gather and Scatter operations within the dataflow. We compare it with the state-of-the-art sparse inference engine TorchSparse 2.0 to show our proposed method can achieve 1.2× speedup for MinkUnet and SPVCNN, without a significant accuracy loss. In particular, our SIMD-based data movement can achieve more than 5× speedup.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: 3D Shape Modeling and Analysis · 3D Surveying and Cultural Heritage · Remote Sensing and LiDAR Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Integrated Circuit Science and Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210023, China
Shenzhen Index Technology Co., Ltd., Shenzhen 518118, China
Funding
National Natural Science Foundation of China · Fundamental Research Funds for the Central Universities · Natural Science Research Foundation of Jiangsu Province for Youth
References (30)
Show source-provided references
1. Advancements in point cloud data augmentation for deep learning: A survey10.1016/j.patcog.2024.110532 ↗
2. Segmentation of 3D point cloud data representing full human body geometry: A review10.1016/j.patcog.2023.109444 ↗
3. Voxel-based segmentation of 3D point clouds from construction sites using a probabilistic connectivity model10.1016/j.patrec.2017.12.016 ↗
4. Bhattacharyya, P., and Czarnecki, K. (2020). Deformable PV-RCNN: Improving 3D object detection with learned deformations. arXiv.
5. Xu, S., Zhou, D., Fang, J., Yin, J., Bin, Z., and Zhang, L. (2021, January 19–22). Fusionpainting: Multimodal fusion with adaptive attention for 3D object detection. Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA.10.1109/ITSC48978.2021.9564951 ↗
6. Multimodal virtual point 3D detection
7. Spatial pruned sparse convolution for efficient 3D object detection
8. Chen, Y., Li, Y., Zhang, X., Sun, J., and Jia, J. (2022, January 18–24). Focal sparse convolutional networks for 3D object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00535 ↗
9. Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21–26). Pointnet: Deep learning on point sets for 3D classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
10. Li, Y., Bu, R., Sun, M., Wu, W., Di, X., and Chen, B. (2018). Pointcnn: Convolution on x-transformed points. Adv. Neural Inf. Process. Syst., 31.
11. Qi, C.R., Su, H., Nießner, M., Dai, A., Yan, M., and Guibas, L.J. (2016, January 27–30). Volumetric and multi-view cnns for object classification on 3D data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.609 ↗
12. Zhou, Y., and Tuzel, O. (2018, January 18–22). Voxelnet: End-to-end learning for point cloud based 3D object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00472 ↗
13. Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., and Li, H. (2020, January 14–19). Pv-rcnn: Point-voxel feature set abstraction for 3D object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01054 ↗
14. Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., and Li, H. (2021, January 2–9). Voxel r-cnn: Towards high performance voxel-based 3D object detection. Proceedings of the AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v35i2.16207 ↗
15. Yin, T., Zhou, X., and Krahenbuhl, P. (2021, January 20–25). Center-based 3D object detection and tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01161 ↗
16. Choy, C., Gwak, J., and Savarese, S. (2019, January 16–17). 4d spatio-temporal convnets: Minkowski convolutional neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00319 ↗
17. Contributors, S. (2025, March 15). Spconv: Spatially Sparse Convolution Library. Available online: https://github.com/traveller59/spconv.
18. Tang, H., Yang, S., Liu, Z., Hong, K., Yu, Z., Li, X., Dai, G., Wang, Y., and Han, S. (November, January 28). Torchsparse++: Efficient training and inference framework for sparse convolution on gpus. Proceedings of the IEEE/ACM International Symposium on Microarchitecture (MICRO), Toronto, ON, Canada.
19. Torchsparse: Efficient point cloud inference engine
20. Graham, B., Engelcke, M., and Van Der Maaten, L. (2018, January 18–22). 3D semantic segmentation with submanifold sparse convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00961 ↗
21. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13–19). nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01164 ↗
22. Dong, X., Huang, J., Yang, Y., and Yan, S. (2017, January 21–26). More is less: A more complicated network with less inference complexity. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.205 ↗
23. Ren, M., Pokrovsky, A., Yang, B., and Urtasun, R. (2018, January 18–22). Sbnet: Sparse blocks network for fast inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00908 ↗
24. Verelst, T., and Tuytelaars, T. (2020, January 13–19). Dynamic convolutions: Exploiting spatial sparsity for faster inference. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00239 ↗
25. Xie, Z., Zhang, Z., Zhu, X., Huang, G., and Lin, S. (2020, January 23–28). Spatially adaptive inference with stochastic feature sampling and interpolation. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part I 16.
26. Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., and Han, S. (2020, January 23–28). Searching efficient 3D architectures with sparse point-voxel convolution. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58604-1_41 ↗
27. Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., and Gall, J. (November, January 27). Semantickitti: A dataset for semantic scene understanding of lidar sequences. Proceedings of the IEEE/CVF international Conference on Computer Vision, Seoul, Repiblic of Korea.
28. Multi-scale interaction for real-time lidar data segmentation on an embedded platform10.1109/LRA.2021.3132059 ↗
29. Zhang, Y., Zhou, Z., David, P., Yue, X., Xi, Z., Gong, B., and Foroosh, H. (2020, January 13–19). Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00962 ↗
30. Xu, C., Wu, B., Wang, Z., Zhan, W., Vajda, P., Keutzer, K., and Tomizuka, M. (2020, January 23–28). Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XXVIII 16.10.1007/978-3-030-58604-1_1 ↗