3D segmentation method for stone cultural relics utilizing multi-modal feature enhancement based on SAM
Source abstract
Abstract
Abstract To enhance segmentation in 3D reconstruction, especially for dense textures, blurred edges, and small structures, this study proposes MF-ESG, a non-invasive method for multi-view image acquisition and 3D segmentation. It introduces the SAGa framework, which combines 3DGS with SAM’s zero-shot capabilities to produce low-dimensional masks from 2D images, guiding multimodal 3D feature learning. This improves adaptability across various stone relic scenes. The edge-sensitive MLLS module, incorporating multi-scale large-kernel attention and dynamic receptive fields, enhances edge segmentation in complex environments. On SPIn-NeRF, NVOS, and custom datasets, the method achieves average mIoU scores of 94.1%, 94.3%, and 95.75%, surpassing SAGa by 0.7%, 1.7%, and 1.23%, respectively. It also significantly reduces edge defects and central voids in reconstruction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: 3D Surveying and Cultural Heritage · Image Processing and 3D Reconstruction · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (31)
Show source-provided references
1. Sun, B. et al. Research on the variations in the weathering of large open-air stone relics from macro- and microperspectives: a case study of the Leshan Giant Buddha. npj Herit. Sci. 13, 102, https://doi.org/10.1038/s40494-025-01566-5 (2025).10.1038/s40494-025-01566-5 ↗
2. Yu, P. K. Cultural relics, intellectual property, and intangible heritage. Temp. L. Rev. 81, 433 (2008).
3. Bin Yan. Overview of the preservation of stone monuments. Southeastern Culture, 90–92 https://doi.org/10.17226/514 (2005).10.17226/514 ↗
4. Meng, T. et al. Highly sensitive terahertz non-destructive testing technology for stone relics deterioration prediction using SVM-based machine learning models. Herit. Sci. 9, 24. https://doi.org/10.1186/s40494-021-00502-7 (2021).10.1186/s40494-021-00502-7 ↗
5. Wang, G., Ye, J. C., Mueller, K. & Fessler, J. A. Image reconstruction is a new frontier of machine learning. IEEE Trans. Med. imaging 37, 1289–1296, https://doi.org/10.1109/TMI.2018.2833635 (2018).10.1109/TMI.2018.2833635 ↗
6. Zhao, C., Huang, X., Yang, K., Wang, X. & Wang, Q. Generalizable 3D Gaussian Splatting for novel view synthesis. Pattern Recognition 161 https://doi.org/10.1016/j.patcog.2024.111271 (2024).10.1016/j.patcog.2024.111271 ↗
7. Kerbl, B., Kopanas, G., Leimkühler, T. & Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42, 139:131-139:114 (2023).10.1145/3592433 ↗
8. Mildenhall, B. et al. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 99–106, https://doi.org/10.48550/arXiv.2003.08934 (2021).10.48550/arXiv.2003.08934 ↗
9. Rother, C., Kolmogorov, V. & Blake, A. GrabCut” interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. ((TOG)) 23, 309–314, https://doi.org/10.1145/1015706.1015720 (2004).10.1145/1015706.1015720 ↗
10. Jiang, J., He, X., Wang, W., Lu, H. & Liu, J. Hierarchical Contrastive Learning for Semantic Segmentation. IEEE Trans. Neural Netw. Learn. Syst. 36, 11202–11214, https://doi.org/10.1109/TNNLS.2024.3491782 (2025).10.1109/TNNLS.2024.3491782 ↗
11. Cen, J. et al. Segment Anything in 3D with Radiance Fields. Int. J. Comput. Vis. 133, 5138–5160, https://doi.org/10.1007/s11263-025-02421-7 (2025).10.1007/s11263-025-02421-7 ↗
12. Archit, A. et al. Segment Anything for Microscopy. Nat. Methods 22, 579–591, https://doi.org/10.1038/s41592-024-02580-4 (2025).10.1038/s41592-024-02580-4 ↗
13. Zou, X. et al. Segment everything everywhere all at once. Adv. Neural Inf. Process. Syst. 36, 19769–19782, https://doi.org/10.48550/arXiv.2304.06718 (2024).10.48550/arXiv.2304.06718 ↗
14. Papanastasiou, G., Dikaios, N., Huang, J., Wang, C. & Yang, G. Is Attention all You Need in Medical Image Analysis? A Review. IEEE J. Biomed. Health Inform. 28, 1398–1411, https://doi.org/10.1109/JBHI.2023.3348436 (2024).10.1109/JBHI.2023.3348436 ↗
15. Wu, Y. et al. Recognition of dispersed organic matter macerals using YOLOv5m model with convolutional block attention module. Fuel 378, 132899. https://doi.org/10.1016/j.fuel.2024.132899 (2024).10.1016/j.fuel.2024.132899 ↗
16. Chen, F., Wei, J., Xue, B. & Zhang, M. Feature fusion and kernel selective in Inception-v4 network. Appl. Soft Comput. 119, 108582. https://doi.org/10.1016/j.asoc.2022.108582 (2022).10.1016/j.asoc.2022.108582 ↗
17. Song, X., Pang, X., Zhang, L., Lu, X. & Hei, X. Single image super-resolution with lightweight multi-scale dilated attention network. Appl. Soft Comput. 169, 112569. https://doi.org/10.1016/j.asoc.2024.112569 (2025).10.1016/j.asoc.2024.112569 ↗
18. Tian, Y. & Bai, K. End-to-End Multitask Learning With Vision Transformer. IEEE Trans. Neural Netw. Learn. Syst. 35, 9579–9590, https://doi.org/10.1109/TNNLS.2023.3234166 (2024).10.1109/TNNLS.2023.3234166 ↗
19. Cheng, G., Han, J. & Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE 105, 1865–1883, https://doi.org/10.1109/JPROC.2017.2675998 (2017).10.1109/JPROC.2017.2675998 ↗
20. Qi, C. R., Yi, L., Su, H. & Guibas, L. J. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Adv. Neural Inform. Process. Sys. 30 https://doi.org/10.5555/3295222.3295263 (2017).10.5555/3295222.3295263 ↗
21. Bu, L. et al. Deep hierarchical learning on point clouds in feature space. Neurocomputing 630, 129647. https://doi.org/10.1016/j.neucom.2025.129647 (2025).10.1016/j.neucom.2025.129647 ↗
22. Yang, B. et al. Learning object bounding boxes for 3d instance segmentation on point clouds. Adv. Neural Inform. Process. Sys. 32, 6737–746, https://doi.org/10.48550/arXiv.1906.01140 (2019).10.48550/arXiv.1906.01140 ↗
23. Rahman, A. U. & Hoskere, V. Instance segmentation of reinforced concrete bridge point clouds with transformers trained exclusively on synthetic data. Autom. Constr. 173, 106067. https://doi.org/10.1016/j.autcon.2025.106067 (2025).10.1016/j.autcon.2025.106067 ↗
24. Liao, G., Zhou, K., Bao, Z., Liu, K. & Li, Q. OV-NeRF: Open-Vocabulary Neural Radiance Fields With Vision and Language Foundation Models for 3D Semantic Understanding. IEEE Trans. Cir. Sys. Video Technol. 34, 12923–12936, https://doi.org/10.1109/tcsvt.2024.3439737 (2024).10.1109/tcsvt.2024.3439737 ↗
25. Huang, R. et al. Joint representation learning for text and 3D point cloud. Pattern Recognit. 147, 110086. https://doi.org/10.1016/j.patcog.2023.110086 (2024).10.1016/j.patcog.2023.110086 ↗
26. Zhang, X., Li, L. & Han, L. A cross-spatial network based on efficient multi-scale attention for landslide recognition. Landslides 21, 2913–2925, https://doi.org/10.1007/s10346-024-02323-8 (2024).10.1007/s10346-024-02323-8 ↗
27. WANG, J. et al. Contextual and selective attention networks for image captioning. SCIENCE CHINA Inf. Sci. 65, 222103. https://doi.org/10.1007/s11432-020-3523-6 (2022).10.1007/s11432-020-3523-6 ↗
28. Sarıgül, M., Ozyildirim, B. M. & Avci, M. Differential convolutional neural network. Neural Netw. 116, 279–287, https://doi.org/10.1016/j.neunet.2019.04.025 (2019).10.1016/j.neunet.2019.04.025 ↗
29. Ruby, U. & Yendapalli, V. Binary cross entropy with deep learning technique for image classification. Int. J. Adv. Trends Comput. Sci. Eng 9 https://doi.org/10.30534/ijatcse/2020/175942020 (2020).10.30534/ijatcse/2020/175942020 ↗
30. Liu, J. & Yu, Z. SA3D-L: A lightweight model for 3D object segmentation using neural radiance fields. Neurocomputing 623, 129420. https://doi.org/10.1016/j.neucom.2025.129420 (2025).10.1016/j.neucom.2025.129420 ↗
31. Mildenhall, B. et al. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Trans. Graph. 38, 1–14 (2019).10.1145/3306346.3322980 ↗