Multi-Scale Upsampling GAN Based Hole-Filling Framework for High-Quality 3D Cultural Heritage Artifacts
Source abstract
Abstract
With the rapid development of 3D scanners, the cultural heritage artifacts can be stored as a point cloud and displayed through the Internet. However, due to natural and human factors, many cultural relics had some surface damage when excavated. As a result, the holes caused by these damages still exist in the generated point cloud model. This work proposes a multi-scale upsampling GAN (MU-GAN) based framework for completing these holes. Firstly, a 3D mesh model based on the original point cloud is reconstructed, and the method of detecting holes is presented. Secondly, the point cloud patch contains hole regions and is extracted from the point cloud. Then the patch is input into the MU-GAN to generate a high-quality dense point cloud. Finally, the empty areas on the original point cloud are filled with the generated dense point cloud patches. A series of real-world experiments are conducted on real scan data to demonstrate that the proposed framework can fill the holes of 3D heritage models with grained details. We hope that our work can provide a useful tool for cultural heritage protection.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: 3D Shape Modeling and Analysis · 3D Surveying and Cultural Heritage · Image Processing and 3D Reconstruction
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, Northwest University, Xi’an 710127, China
National and Local Joint Engineering Research Center for Cultural Heritage Digitization, Xi’an 710127, China
Funding
State Key Program of National Natural Science Foundation of China · Key industrial chain projects in Shaanxi Province · National Key Research and Development Program of China · China Postdoctoral Science Foundation · Young Talent Support Program of the Shaanxi Association for Science and Technology · Key Research and Development Program of Shaanxi Province · Major research and development project of Qinghai
References (42)
Show source-provided references
1. Filling holes with shape preserving conditions10.1016/j.matcom.2014.12.008 ↗
2. Kazhdan, M., Bolitho, M., and Hoppe, H. (2006, January 26–28). Poisson surface reconstruction. Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Cagliari, Italy.
3. Filling holes in digitized point cloud using a morphing-based approach to preserve volume characteristics10.1007/s00170-015-7185-0 ↗
4. Automatic hole-filling of CAD models with feature-preserving10.1016/j.cag.2011.12.007 ↗
5. A robust hole-filling algorithm for triangular mesh10.1007/s00371-007-0167-y ↗
6. A lightweight approach to repairing digitized polygon meshes10.1007/s00371-010-0416-3 ↗
7. Hole filling of a 3D model by flipping signs of a signed distance field in adaptive resolution10.1109/TPAMI.2007.70726 ↗
8. Sahay, P., and Rajagopalan, A. (2015, January 7–12). Geometric inpainting of 3D structures. Proceedings of the Computer Vision and Patten Recognition, Boston, MA, USA.10.1109/CVPRW.2015.7301388 ↗
9. A Sharpness-Dependent Filter for Recovering Sharp Features in Repaired 3D Mesh Models10.1109/TVCG.2007.70625 ↗
10. Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., and Ranzuglia, G. (2008, January 2–4). Meshlab: An open-source mesh processing tool. Proceedings of the Eurographics Italian Chapter Conference, Salerno, Italy.
11. Elshishiny, H.E.E.D., Bernardini, F., and Rushmeier, H.E. System and Method for Hole Filling in 3D Models. (7,272,264), U.S. Patent.
12. Context-based coherent surface completion10.1145/2532548 ↗
13. Pauly, M., Mitra, N.J., Wallner, J., Pottmann, H., and Guibas, L.J. (2008, January 11–15). Discovering structural regularity in 3D geometry. Proceedings of the SIGGRAPH’08: International Conference on Computer Graphics and Interactive Techniques, Dallas, TX, USA.10.1145/1399504.1360642 ↗
14. Fu, Z., Hu, W., and Guo, Z. (2018, January 7–10). Point cloud inpainting on graphs from non-local self-similarity. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451550 ↗
15. Filling holes with geometric and volumetric constraints10.1016/j.camwa.2017.05.009 ↗
16. Multiple-gamma-value based phase error compensation method for phase measuring profilometry10.1364/AO.57.010290 ↗
17. Hoang, V.-D., Le, M.-H., Hernández, D.C., and Jo, K.-H. (2013, January 10–13). Localization estimation based on Extended Kalman filter using multiple sensors. Proceedings of the IECON 2013—39th Annual Conference of the IEEE Industrial Electronics Society, Vienna, Austria.10.1109/IECON.2013.6700032 ↗
18. Hole-filling algorithm based on poisson equation
19. Ngo, T.M., and Lee, W.S. (2013, January 21–24). Feature-First Hole Filling Strategy for 3D Meshes. Proceedings of the Computer Vision, Imaging and Computer Graphics, Theory and Applications, Barcelona, Spain.10.1007/978-3-642-32350-8_4 ↗
20. A novel dual-camera calibration method for 3D optical measurement10.1016/j.optlaseng.2017.09.025 ↗
21. Vokhmintcev, A., Timchenko, M., and Alina, K. (2017, January 16–19). Real-time visual loop-closure detection using fused iterative close point algorithm and extended Kalman filter. Proceedings of the 2017 International Conference on Industrial Engineering, Applications and Manufacturing (ICIEAM), Saint Petersburg, Russia.10.1109/ICIEAM.2017.8076187 ↗
22. Research on a hole filling algorithm of a point cloud based on structure from motion10.1364/JOSAA.36.000A39 ↗
23. Dai, A., Ruizhongtai Qi, C., and Nießner, M. (2017, January 21–26). Shape completion using 3d-encoder-predictor cnns and shape synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.693 ↗
24. Nguyen, D.T., Hua, B.-S., Tran, K., Pham, Q.-H., and Yeung, S.-K. (2016, January 27–30). A field model for repairing 3d shapes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.612 ↗
25. Learning 3d shape completion under weak supervision10.1007/s11263-018-1126-y ↗
26. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. (2015, January 7–12). 3d shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
27. Han, X., Li, Z., Huang, H., Kalogerakis, E., and Yu, Y. (2017, January 21–26). High-resolution shape completion using deep neural networks for global structure and local geometry inference. Proceedings of the IEEE International Conference on Computer Vision, Honolulu, HI, USA.10.1109/ICCV.2017.19 ↗
28. Sharma, A., Grau, O., and Fritz, M. (2016, January 8–14). Vconv-dae: Deep volumetric shape learning without object labels. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-319-49409-8_20 ↗
29. Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21–26). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
30. Fan, H., Su, H., and Guibas, L.J. (2017, January 21–26). A point set generation network for 3d object reconstruction from a single image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.264 ↗
31. Huang, Q., Wang, W., and Neumann, U. (2018, January 18–22). Recurrent slice networks for 3d segmentation of point clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00278 ↗
32. Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017, January 4–9). Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
33. Achlioptas, P., Diamanti, O., Mitliagkas, I., and Guibas, L. (2017, January 6–11). Learning Representations and Generative Models for 3D Point Clouds. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
34. Yang, Y., Feng, C., Shen, Y., and Tian, D. (2018, January 18–22). Foldingnet: Point cloud auto-encoder via deep grid deformation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00029 ↗
35. Yuan, W., Khot, T., Held, D., Mertz, C., and Hebert, M. (2018, January 5–8). Pcn: Point completion network. Proceedings of the 2018 International Conference on 3D Vision (3DV), Verona, Italy.10.1109/3DV.2018.00088 ↗
36. Chen, X., Chen, B., and Mitra, N.J. (2019). Unpaired point cloud completion on real scans using adversarial training. arXiv.
37. Sarmad, M., Lee, H.J., and Kim, Y.M. (2019, January 16–17). Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00605 ↗
38. Yu, L., Li, X., Fu, C.-W., Cohen-Or, D., and Heng, P.-A. (2018, January 18–22). Pu-net: Point cloud upsampling network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00295 ↗
39. Yifan, W., Wu, S., Huang, H., Cohen-Or, D., and Sorkine-Hornung, O. (2019, January 16–17). Patch-based progressive 3d point set upsampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00611 ↗
40. Li, R., Li, X., Fu, C.-W., Cohen-Or, D., and Heng, P.-A. (2019, January 16–17). Pu-gan: A point cloud upsampling adversarial network. Proceedings of the IEEE/CVF International Conference on Computer Vision, Long Beach, CA, USA.10.1109/ICCV.2019.00730 ↗
41. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
42. Binary volumetric convolutional neural networks for 3-D object recognition10.1109/TIM.2018.2840598 ↗