GSV-NET: A Multi-Modal Deep Learning Network for 3D Point Cloud Classification
Source abstract
Abstract
Light Detection and Ranging (LiDAR), which applies light in the formation of a pulsed laser to estimate the distance between the LiDAR sensor and objects, is an effective remote sensing technology. Many applications use LiDAR including autonomous vehicles, robotics, and virtual and augmented reality (VR/AR). The 3D point cloud classification is now a hot research topic with the evolution of LiDAR technology. This research aims to provide a high performance and compatible real-world data method for 3D point cloud classification. More specifically, we introduce a novel framework for 3D point cloud classification, namely, GSV-NET, which uses Gaussian Supervector and enhancing region representation. GSV-NET extracts and combines both global and regional features of the 3D point cloud to further enhance the information of the point cloud features for the 3D point cloud classification. Firstly, we input the Gaussian Supervector description into a 3D wide-inception convolution neural network (CNN) structure to define the global feature. Secondly, we convert the regions of the 3D point cloud into color representation and capture region features with a 2D wide-inception network. These extracted features are inputs of a 1D CNN architecture. We evaluate the proposed framework on the point cloud dataset: ModelNet and the LiDAR dataset: Sydney. The ModelNet dataset was developed by Princeton University (New Jersey, United States), while the Sydney dataset was created by the University of Sydney (Sydney, Australia). Based on our numerical results, our framework achieves more accuracy than the state-of-the-art approaches.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Remote Sensing and LiDAR Applications · 3D Shape Modeling and Analysis · 3D Surveying and Cultural Heritage
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence Convergence, Pukyong National University, Busan 48513, Korea
Department of Computer Engineering, Dong-A University, Busan 49315, Korea
Division of Artificial Intelligence, Tongmyong University, Busan 48520, Korea
References (62)
Show source-provided references
1. Stereo matching using multi-level cost volume and multi-scale feature constancy10.1109/TPAMI.2019.2928550 ↗
2. Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., and Bennamoun, M. (2020). Deep learning for 3d point clouds: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 4338–4364.10.1109/TPAMI.2020.3005434 ↗
3. Rotational projection statistics for 3D local surface description and object recognition10.1007/s11263-013-0627-y ↗
4. 3D object recognition in cluttered scenes with local surface features: A survey10.1109/TPAMI.2014.2316828 ↗
5. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-View 3D object detection network for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
6. Point cloud classification model based on a dual-input deep network framework10.1109/ACCESS.2020.2981357 ↗
7. Chen, B., Shi, S., Gong, W., Zhang, Q., Yang, J., Du, L., Sun, J., Zhang, Z., and Song, S. (2017). Multispectral LiDAR point cloud classification: A two-step approach. Remote Sens., 9.10.3390/rs9040373 ↗
8. Maes, W., Huete, A., and Steppe, K. (2017). Optimizing the processing of UAVbased thermal imagery. Remote Sens., 9.10.3390/rs9050476 ↗
9. A multiscale and hierarchical feature extraction method for terrestrial laser scanning point cloud classification10.1109/TGRS.2014.2359951 ↗
10. Xie, Y., Tian, J., and Zhu, X.X. (2019). A review of point cloud semantic segmentation. arXiv.
11. Griffiths, D., and Boehm, J. (2019). A review on deep learning techniques for 3D sensed data classification. Remote Sens., 11.10.3390/rs11121499 ↗
12. Contextual segment-based classification of airborne laser scanner data10.1016/j.isprsjprs.2017.03.010 ↗
13. A structured regularization framework for spatially smoothing semantic labelings of 3D point clouds10.1016/j.isprsjprs.2017.08.010 ↗
14. A review of point clouds segmentation and classification algorithms
15. 3D shape recognition based on multi-modal information fusion10.1007/s11042-019-08552-7 ↗
16. Point cloud classification with deep normalized Reeb graph convolution10.1016/j.imavis.2020.104092 ↗
17. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J., and Fisher, Y. (2015, January 7–12). 3D ShapeNets: A deep representation for volumetric shapes. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
18. Maturana, D., and Scherer, S. (October, January 28). VoxNet: A 3D convolutional neural network for real-time object recognition. Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.10.1109/IROS.2015.7353481 ↗
19. Riegler, G., Ulusoy, A.O., and Geiger, A. (2017, January 21–26). Octnet: Learning deep 3D representations at high resolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.701 ↗
20. 3DmFV: Three-dimensional point cloud classification in real-time using convolutional neural networks
21. Le, T., and Duan, Y. (2018, January 18–23). Pointgrid: A deep network for 3D shape understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00959 ↗
22. Yang, Z., and Wang, L. (November, January 7). Learning relationships for multi-view 3D object recognition. Proceedings of the 2019 IEEE International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00760 ↗
23. Yu, T., Meng, J., and Yuan, J. (2018, January 18–23). Multi-view harmonized bilinear network for 3D object recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00027 ↗
24. Su, H., Maji, S., Kalogerakis, E., and Learned-Miller, E. (2015, January 7–13). Multi-view convolutional neural networks for 3D shape recognition. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.114 ↗
25. Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21–26). Pointnet: Deep learning on point sets for 3D classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
26. Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017). PointNet++: Deep hierarchical feature learning on point sets in a metric space. arXiv.
27. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
28. Drui, F., Franck, E., Helluy, P., and Navoret, L. (2018). An analysis of overrelaxation in kinetic approximation. arXiv.10.1016/j.crme.2018.12.001 ↗
29. PointCNN: Convolution on X-transformed points
30. Chen, S., Tian, D., Feng, C., Vetro, A., and Kovacevic, J. (2017, January 5–9). Contour-enhanced resampling of 3D point clouds via graphs. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952695 ↗
31. Chen, S., Tian, D., Feng, C., Vetro, A., and Kovačević, J. (2017). Fast resampling of 3d point clouds via graphs. arXiv.10.1109/ICASSP.2017.7952695 ↗
32. PDE-based graph signal processing for 3-D color point clouds: Opportunities for cultural heritage10.1109/MSP.2015.2408631 ↗
33. Graph-based compression of dynamic 3D point cloud sequences10.1109/TIP.2016.2529506 ↗
34. Defferrard, M., Bresson, X., and Vandergheynst, P. (2016, January 4–9). Convolutional neural networks on graphs with fast localized spectral filtering. Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS’16), Barcelona, Spain.
35. Bruna, J., Zaremba, W., Szlam, A., and Lecun, Y. (2013). Spectral networks and locally connected networks on graphs. arXiv.
36. Zhang, Y., and Rabbat, M. (2018). A graph-CNN for 3D point cloud classification. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE.10.1109/ICASSP.2018.8462291 ↗
37. 1D convolutional neural networks and applications: A survey10.1016/j.ymssp.2020.107398 ↗
38. A comparison of Fisher vectors and Gaussian Supervectors for document versus non-document image classification10.1117/12.2023329 ↗
39. Novel Gaussianized vector representation for improved natural scene categorization10.1016/j.patrec.2009.12.010 ↗
40. Kang, G.X., Liu, K., Hou, B.B., and Zhang, N. (2017). 3D multi-view convolutional neural networks for lung nodule classification. PLoS ONE, 12.10.1371/journal.pone.0188290 ↗
41. Muhammad, W., and Aramvith, S. (2019). Multi-scale inception based super-resolution using deep learning approach. Electronics, 8.10.3390/electronics8080892 ↗
42. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
43. He, K., and Sun, J. (2015, January 7–12). Convolutional neural networks at constrained time cost. Proceedings of the CVPR, Boston, MA, USA.10.1109/CVPR.2015.7299173 ↗
44. Zagoruyko, S., and Komodakis, N. (2016). Wide Residual Networks. arXiv.10.5244/C.30.87 ↗
45. Lee, Y., Kim, H., Park, E., Cui, X., and Kim, H. (2017, January 11–14). Wide-residual-inception networks for real-time object detection. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.10.1109/IVS.2017.7995808 ↗
46. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 8–10). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
47. Kandel, I., and Castelli, M. (2020). Transfer learning with convolutional neural networks for diabetic retinopathy image classification. A review. Appl. Sci., 10.10.3390/app10062021 ↗
48. Hoang, H.H., and Trinh, H.H. (2021). Improvement for Convolutional Neural Networks in Image Classification Using Long Skip Connection. Appl. Sci., 11.10.3390/app11052092 ↗
49. Quadros, A.J. (2013). Representing 3D Shape in Sparse Range Images for Urban Object Classification. [Ph.D. Thesis, The University of Sydney]. Available online: http://www.acfr.usyd.edu.au/papers/SydneyUrbanObjectsDataset.shtml.
50. Deuge, M.D., Quadros, A., Hung, C., and Douillard, B. (2013, January 2–4). Unsupervised feature learning for classification of outdoor 3D scans. Proceedings of Australasian Conference on Robotics and Automation, Sydney, Australia. Available online: https://www.araa.asn.au/acra/acra2013/papers/pap133s1-file1.pdf.
51. Learning high-level features by fusing multi-view representation of MLS point clouds for 3D object recognition in road environments10.1016/j.isprsjprs.2019.01.024 ↗
52. Seo, K., Chung, B., Panchaseelan, H.P., Kim, T., Park, H., Oh, B., Chun, M., Won, S., Kim, D., and Beom, J. (2021). Forecasting the Walking Assistance Rehabilitation Level of Stroke Patients Using Artificial Intelligence. Diagnostics, 11.10.3390/diagnostics11061096 ↗
53. Ren, M., Niu, L., and Fang, Y. (2017). 3D-A-Nets: 3D deep dense descriptor for volumetric shapes with adversarial networks. arXiv.
54. An effective encoding method based on local information for 3D point cloud classification10.1109/ACCESS.2019.2905595 ↗
55. Zhang, L., Sun, J., and Zheng, Q. (2018). 3D point cloud recognition based on a multi-view convolutional neural network. Sensors, 18.10.3390/s18113681 ↗
56. Han, X.F., Sun, S.J., Song, X.Y., and Xiao, G.Q. (2018). 3D point cloud descriptors in hand-crafted and deep learning age: State-of-the-art. arXiv.
57. Munoz, D., Bagnell, J.A., and Hebert, M. (2012, January 7–13). Co-inference for multi-modal scene analysis. Proceedings of the European Conference on Computer Vision (ECCV), Florence, Italy.10.1007/978-3-642-33783-3_48 ↗
58. Gupta, A. (2020). Deep Learning for Semantic Feature Extraction in Aerial Imagery and LiDAR Data. [Ph.D. Thesis, University of Manchester]. Available online: https://www.research.manchester.ac.uk/portal/files/184627877/FULL_TEXT.PDF.
59. Binary volumetric convolutional neural networks for 3-D object recognition10.1109/TIM.2018.2840598 ↗
60. NormalNet: A voxel-based CNN for 3D object classification and retrieval10.1016/j.neucom.2018.09.075 ↗
61. Sedaghat, N., Zolfaghari, M., Amiri, E., and Brox, T. (2017, January 4–7). Orientation-boosted voxel nets for 3D object recognition. Proceedings of the 28th British Machine Vision Conference, London, UK.10.5244/C.31.97 ↗
62. Yoo, I. (2021, August 22). Point Cloud Deep Learning. Available online: On-demand.gputechconf.com/gtc/2018/presentation/s8453-point-cloud-deep-learning.pdf.