Cross modal networks for point cloud semantic segmentation of Chinese ancient buildings
Source abstract
Abstract
Point clouds have become essential for digital heritage preservation, aiding in the identification and classification of complex structural elements. However, most datasets rely on single-modal data, limiting their ability to describe real-world scenarios comprehensively. This paper introduces the Real-World Multi-modal Ancient Architecture Point Cloud Semantic Segmentation Dataset (RW-MAPCSD), which includes multi-modal data such as point clouds, line drawings, color, and depth projections, enabling multi-modal analysis of ancient buildings. To address data imbalance, we propose a novel segmentation network, Mask2former-KNN 3D Network (MK3DNet). The network projects point clouds into images while preserving point indices, using image segmentation techniques for initial segmentation. Results are then mapped back to the point cloud and refined with the K-nearest neighbors (KNN) algorithm. Experimental results show significant improvements, with mIoU and OA of 77.47% and 90.85%, respectively, surpassing the Point Transformer network by 21.94% and 5.87%.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: 3D Surveying and Cultural Heritage · Image Processing and 3D Reconstruction · Remote Sensing and LiDAR Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (67)
Show source-provided references
1. Liu, J. & Wu, Z. K. Rule-based generation of ancient Chinese architecture from the song dynasty. J. Comput. Cultural Herit. (JOCCH) 9, 1–22, https://doi.org/10.1145/2835495 (2015).10.1145/2835495 ↗
2. Hu, Q. et al. Fine surveying and 3D modeling approach for wooden ancient architecture via multiple laser scanner integration. Remote Sens. 8, 270, https://doi.org/10.3390/rs8040270 (2016).10.3390/rs8040270 ↗
3. Biryukova, M. V. & Nikonova, A. A. The role of digital technologies in the preservation of cultural heritage. Muzeol.ógia a kult.úrne dedičstvo 5, 1 (2017).
4. Adane, A., Chekole, A. & Gedamu, G. Cultural heritage digitization: Challenges and opportunities. Int. J. Computer Appl. 178, 1–5, https://doi.org/10.5120/ijca2019919180 (2019).10.5120/ijca2019919180 ↗
5. Hu, Y. et al. Measurement and analysis of facial features of terracotta warriors based on high-precision 3D point clouds. Herit. Sci. 10, 40, https://doi.org/10.1186/s40494-022-00662-0 (2022).10.1186/s40494-022-00662-0 ↗
6. Lague, D., Brodu, N. & Leroux, J. Accurate 3D comparison of complex topography with terrestrial laser scanner: Application to the Rangitikei canyon (N-Z). ISPRS J. Photogramm. Remote Sens. 82, 10–26, https://doi.org/10.1016/j.isprsjprs.2013.04.009 (2013).10.1016/j.isprsjprs.2013.04.009 ↗
7. Guo, Y. et al. Deep learning for 3D point clouds: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 43, 4338–4364, https://doi.org/10.1109/TPAMI.2020.3005434 (2020).10.1109/TPAMI.2020.3005434 ↗
8. Haznedar, B., Bayraktar, R., Ozturk, A. E. & Arayici, Y. Implementing PointNet for point cloud segmentation in the heritage context. Herit. Sci. 11, 2, https://doi.org/10.1186/s40494-022-00844-w (2023).10.1186/s40494-022-00844-w ↗
9. Armeni, I. et al. 3d semantic parsing of large-scale indoor spaces. Proceedings of the IEEE conference on computer vision and pattern recognition;1534-1543. https://doi.org/10.1109/CVPR.2016.170. (2016)10.1109/CVPR.2016.170 ↗
10. Hackel, T. et al. Semantic3D.net: A new Large-scale Point Cloud Classification Benchmark. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences. https://doi.org/10.5194/isprs-annals-IV-1-W1-91-2017 (2017).10.5194/isprs-annals-IV-1-W1-91-2017 ↗
11. Chen, M. et al. An Aerial Photogrammetry Benchmark Dataset for Point Cloud Segmentation and Style Translation. Remote Sens. 16, 22 (2024).10.3390/rs16224240 ↗
12. Djelliout, T. & Aliane, H. Building and evaluation of an Algerian Cultural Heritage dataset using convolutional neural networks. 2022 4th International Conference on Pattern Analysis and Intelligent Systems (PAIS);1-7. https://doi.org/10.1109/PAIS56586.2022.9946666 (2022).10.1109/PAIS56586.2022.9946666 ↗
13. Swaileh, W. et al. Versailles-FP Dataset: Wall Detection in Ancient Floor Plans. Springer, Cham. https://doi.org/10.1007/978-3-030-86549-8_3 (2021).10.1007/978-3-030-86549-8_3 ↗
14. Barz, B. & Denzler, J. Wikichurches: A fine-grained dataset of architectural styles with real-world challenges. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://doi.org/10.5281/zenodo.5166986 (2021).10.5281/zenodo.5166986 ↗
15. Robot Vision Group. National Laboratory of Pattern Recogntion. Institute of Automation, Chinese Academy of Sciences. 3D Reconstruction Dataset. http://vision.ia.ac.cn/data.
16. Gabara, G. & Sawicki, P. CRBeDaSet: A benchmark dataset for high accuracy close range 3D object reconstruction. Remote Sens. 15, 1116, https://doi.org/10.3390/rs15041116 (2023).10.3390/rs15041116 ↗
17. Matrone, F. et al. A benchmark for large-scale heritage point cloud semantic segmentation. Int. Arch. Photogrammetr. Remote Sens. Spatial Inform. Sci. 43, 1419–1426, https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-1419-2020 (2020).10.5194/isprs-archives-XLIII-B2-2020-1419-2020 ↗
18. Yang, B. S., Han, X. & Dong, Z. Point cloud benchmark dataset WHU-TLS and WHU-MLS for deep learning. Natl Remote Sens. Bull. 25, 231–240, https://doi.org/10.11834/jrs.20210542 (2021).10.11834/jrs.20210542 ↗
19. Zhou, C., Dong, Y. & Hou, M. DGPCD: a benchmark for typical official-style Dougong in ancient Chinese wooden architecture. Herit. Sci. 12, 201, https://doi.org/10.1186/s40494-024-01293-3 (2024).10.1186/s40494-024-01293-3 ↗
20. Grilli, E. & Remondino, F. Machine learning generalisation across different 3D architectural heritage. ISPRS Int. J. Geo-Inf. 9, 379, https://doi.org/10.3390/ijgi9060379 (2020).10.3390/ijgi9060379 ↗
21. Gao, X. et al. Ancient Chinese architecture 3D preservation by merging ground and aerial point clouds. ISPRS J. Photogramm. Remote Sens. 143, 72–84, https://doi.org/10.1016/j.isprsjprs.2018.04.023 (2018).10.1016/j.isprsjprs.2018.04.023 ↗
22. Liu, Y., Gao, W. & Hu, Z. A large-scale dataset for indoor visual localization with high-precision ground truth. Int. J. Robot. Res. 41, 129–135, https://doi.org/10.1177/02783649211052064 (2022).10.1177/02783649211052064 ↗
23. Walczak J. InLUT3D: Challenging real indoor dataset for point cloud analysis. arXiv preprint. 2024; arXiv:2408.03338. https://doi.org/10.48550/arXiv.2408.03338.10.48550/arXiv.2408.03338 ↗
24. Hossain, M. et al. Building Indoor Point Cloud Datasets with Object Annotation for Public Safety. SMARTGREENS; 45–56. https://doi.org/10.5220/0010454400450056 (2021).10.5220/0010454400450056 ↗
25. Ibrahim, M., Akhtar, N., Wise, M. & Mian, A. Annotation tool and urban dataset for 3d point cloud semantic segmentation. IEEE Access 9, 35984–35996, https://doi.org/10.1109/ACCESS.2021.3062547 (2021).10.1109/ACCESS.2021.3062547 ↗
26. Wang R., Huang S., Yang H. Building3d: A. urban-scale dataset and benchmarks for learning roof structures from point clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision; 20076-20086. https://doi.org/10.1109/ICCV51070.2023.01837 (2023).10.1109/ICCV51070.2023.01837 ↗
27. Krapf, S., Mayer, K. & Fischer, M. Points for energy renovation (PointER): A point cloud dataset of a million buildings linked to energy features. Sci. Data 10, 639, https://doi.org/10.1038/s41597-023-02544-x (2023).10.1038/s41597-023-02544-x ↗
28. Pierdicca, R., Mameli, M., Malinverni, E. S., Paolanti, M. & Frontoni, E. Automatic generation of point cloud synthetic dataset for historical building representation. Augmented Reality, Virtual Reality, and Computer Graphics: 6th International Conference, AVR 2019, Santa Maria al Bagno, Italy, June 24–27, 2019, Proceedings, Part I 6. Springer International Publishing; 203–219. https://doi.org/10.1007/978-3-030-25965-5_16 (2019).10.1007/978-3-030-25965-5_16 ↗
29. Battini, C. et al. Automatic generation of synthetic heritage point clouds: Analysis and segmentation based on shape grammar for historical vaults. J. Cultural Herit. 66, 37–47, https://doi.org/10.1016/j.culher.2023.10.003 (2024).10.1016/j.culher.2023.10.003 ↗
30. Morbidoni, C., Pierdicca, R., Paolanti, M., Quattrini, R. & Mammoli, R. Learning from synthetic point cloud data for historical buildings semantic segmentation. J. Comput. Cultural Herit. (JOCCH) 13, 1–16, https://doi.org/10.1145/340926 (2020).10.1145/340926 ↗
31. Albano, R. Investigation on roof segmentation for 3D building reconstruction from aerial LIDAR point clouds. Appl. Sci. 9, 4674, https://doi.org/10.3390/app9214674 (2019).10.3390/app9214674 ↗
32. Hamid-Lakzaeian, F. Structural-based point cloud segmentation of highly ornate building façades for computational modelling. Autom. Constr. 108, 102892, https://doi.org/10.1016/j.autcon.2019.102892 (2019).10.1016/j.autcon.2019.102892 ↗
33. Salamanca, S., Merchán, P., Espacio, A., Pérez, E. & Merchán, M. J. Segmentation of 3D Point Clouds of Heritage Buildings using Edge Detection and Supervoxel-Based Topology. Sens. (Basel) 24, 4390, https://doi.org/10.3390/s24134390 (2024).10.3390/s24134390 ↗
34. Zolanvari, S. M. I., Laefer, D. F. & Natanzi, A. S. Three-dimensional building façade segmentation and opening area detection from point clouds. ISPRS J. Photogramm. Remote Sens. 143, 134–149, https://doi.org/10.1016/j.isprsjprs.2018.04.004 (2018).10.1016/j.isprsjprs.2018.04.004 ↗
35. Presti, N. L. et al. Streamlining FE and BIM Modeling for Historic Buildings with Point Cloud Transformation. International Journal of Architectural Heritage; 1-14. https://doi.org/10.1080/15583058.2024.2330942 (2024).10.1080/15583058.2024.2330942 ↗
36. Tao, W., Xiao, Y., Wang, R., Lu, T. & Xu, S. A Fast Registration Method for Building Point Clouds Obtained by Terrestrial Laser Scanner via 2D Feature Points. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. https://doi.org/10.1109/JSTARS.2024.3392927 (2024).10.1109/JSTARS.2024.3392927 ↗
37. Dong, Y., Hou, M., Xu, B., Li, Y. & Ji, Y. Ming and Qing Dynasty Official-Style Architecture Roof Types Classification Based on the 3D Point Cloud. ISPRS Int. J. Geo-Inf. 10, 650, https://doi.org/10.3390/ijgi10100650 (2021).10.3390/ijgi10100650 ↗
38. Croce, V. et al. From the semantic point cloud to heritage-building information modeling: A semiautomatic approach exploiting machine learning. Remote Sens. 13, 461, https://doi.org/10.3390/rs13030461 (2021).10.3390/rs13030461 ↗
39. Stober, D. & Raguz-Lucic, N. Trends, Problems, and Solutions from Point Cloud via Non-Uniform Rational Basis Spline to Building Information Modelling: Bibliometric and Systematic Study. Buildings 14, 564, https://doi.org/10.3390/buildings14030564 (2024).10.3390/buildings14030564 ↗
40. Wang, H., Shi, Y., Yuan, Q. & Li, M. Crack Detection and Feature Extraction of Heritage Buildings via Point Clouds: A Case Study of Zhonghua Gate Castle in Nanjing. Buildings 14, 2278, https://doi.org/10.3390/buildings14082278 (2024).10.3390/buildings14082278 ↗
41. Zou, Z., Zhao, P. & Zhao, X. Automatic segmentation, inpainting, and classification of defective patterns on ancient architecture using multiple deep learning algorithms. Struct. Control Health Monit. 28, e2742, https://doi.org/10.1002/stc.2742 (2021).10.1002/stc.2742 ↗
42. Cao, Y. & Scaioni, M. 3DLEB-Net: Label-efficient deep learning-based semantic segmentation of building point clouds at LoD3 level. Appl. Sci. 11, 8996, https://doi.org/10.3390/app11198996 (2021).10.3390/app11198996 ↗
43. Xiong, Y., Chen, Q., Zhu, M., Zhang, Y. & Huang, K. Accurate detection of historical buildings using aerial photographs and deep transfer learning. IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium;1592-1595. https://doi.org/10.1109/IGARSS39084.2020.9323541 (2020).10.1109/IGARSS39084.2020.9323541 ↗
44. Zhou, C., Dong, Y., Hou, M., Ji, Y. & Wen, C. MP-DGCNN for the semantic segmentation of Chinese ancient building point clouds. Herit. Sci. 12, 249, https://doi.org/10.1186/s40494-024-01289-z (2024).10.1186/s40494-024-01289-z ↗
45. Zhao, J. et al. DSC-Net: learning discriminative spatial contextual features for semantic segmentation of large-scale ancient architecture point clouds. Herit. Sci. 12, 274, https://doi.org/10.1186/s40494-024-01367-2 (2024).10.1186/s40494-024-01367-2 ↗
46. Zhang, R. et al. MSFA-Net: A Multiscale Feature Aggregation Network for Semantic Segmentation of Historical Building Point Clouds. Buildings 14, 1285, https://doi.org/10.3390/buildings14051285 (2024).10.3390/buildings14051285 ↗
47. Zhao, J. et al. Semantic segmentation of point clouds of ancient buildings based on weak supervision. Herit. Sci. 12, 232, https://doi.org/10.1186/s40494-024-01353-8 (2024).10.1186/s40494-024-01353-8 ↗
48. Xiao, C., Chen, Y., Sun, C., You, L. & Li, R. AM-ESRGAN: Super-Resolution Reconstruction of Ancient Murals Based on Attention Mechanism and Multi-Level Residual Network. Electronics 13, 3142, https://doi.org/10.3390/electronics13163142 (2024).10.3390/electronics13163142 ↗
49. Li, B. et al. Building a Chinese ancient architecture multimodal dataset combining image, annotation and style-model. Sci. Data 11, 1137, https://doi.org/10.1038/s41597-024-03946-1 (2024).10.1038/s41597-024-03946-1 ↗
50. Duan, S., Wang, J. & Su, Q. Restoring Ancient Ideograph: A Multimodal Multitask Neural Network Approach. arXiv preprint. 2024; arXiv:2403.06682. https://doi.org/10.48550/arXiv.2403.06682.10.48550/arXiv.2403.06682 ↗
51. Guo, Q. Yingzao Fashi: Twelfth-century chinese building manual. Architectural Hist. 41, 1–13, https://doi.org/10.2307/1568644 (1998).10.2307/1568644 ↗
52. Li, S. Reconstituting Chinese building tradition: the Yingzao fashi in the early twentieth centur. J. Soc. Architectural Historians 62, 470–489, https://doi.org/10.2307/3592498 (2003).10.2307/3592498 ↗
53. Zhu, X. et al. PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning. Proceedings of the IEEE/CVF International Conference on Computer Vision;2639-2650. http://arxiv.org/abs/2211.11682 (2023).10.1109/ICCV51070.2023.00249 ↗
54. Cover, T. & Hart, P. Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13, 21–27, https://doi.org/10.1109/TIT.1967.1053964 (1967).10.1109/TIT.1967.1053964 ↗
55. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R. Masked-attention mask transformer for universal image segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition;1290-1299. http://arxiv.org/abs/2112.01527 (2022).10.1109/CVPR52688.2022.00135 ↗
56. Chen, L. C., Papandreou, G., Schroff, F. & Adam, H. Rethinking Atrous Convolution for Semantic Image Segmentation. CoRR abs/1706.05587 (2017).
57. Huang, Z. et al. Ccnet: Criss-cross attention for semantic segmentation.Proceedings of the IEEE/CVF international conference on computer vision;03-612. https://doi.org/10.48550/arXiv.1811.11721 (2019).10.48550/arXiv.1811.11721 ↗
58. Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation.oceedings of the IEEE conference on computer vision and pattern recognition;431-3440. https://doi.org/10.1109/CVPR.2015.7298965 (2015).10.1109/CVPR.2015.7298965 ↗
59. Chen, L. C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European conference on computer vision (ECCV);801-818. http://arxiv.org/abs/1802.02611 (2018).10.1007/978-3-030-01234-2_49 ↗
60. Lin, X., Guo, Y. & Wang, J. Global Correlation Network: End-to-End Joint Multi-Object Detection and Tracking. CoRR abs/2103.12511 (2021).
61. Xie, E. et al. SegFormer: Simple and efficient design for semantic segmentation with transformers. Adv. Neural Inf. Process. Syst. 34, 12077–12090 (2021).
62. Hu, Q. et al. RandLA-Net: Efficient semantic segmentation of large-scale point clouds. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR);11108–11117. https://doi.org/10.1109/CVPR42600.2020.01112 (2020).10.1109/CVPR42600.2020.01112 ↗
63. Qi, C. R., Yi, L., Su, H. & Guibas, L. J. PointNet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems;30. http://arxiv.org/abs/1706.02413 (2017).
64. Wang, L., Huang, Y., Hou, Y., Zhang, S. & Shan, J. Graph attention convolution for point cloud semantic segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition;10296–10305. https://doi.org/10.1109/CVPR.2019.01054 (2019).10.1109/CVPR.2019.01054 ↗
65. Liu, Z., Tang, H., Lin, Y. & Han, S. Point-voxel cnn for efficient 3d deep learning. Advances in neural information processing systems;32 (2019).
66. Xu, M., Ding, R., Zhao, H. & Qi, X. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds.Proceedings of the IEEE/CVF conference on computer vision and pattern recognition;3173-3182. https://doi.org/10.1109/CVPR46437.2021.00319 (2021).10.1109/CVPR46437.2021.00319 ↗
67. Engel, N., Belagiannis, V. & Dietmayer, K. Point Transformer. IEEE Access 9, 134826–134840, https://doi.org/10.1109/ACCESS.2021.3116304 (2021).10.1109/ACCESS.2021.3116304 ↗