Classification of Architectural Heritage Images Using Deep Learning Techniques
Source abstract
Abstract
The classification of the images taken during the measurement of an architectural asset is an essential task within the digital documentation of cultural heritage. A large number of images are usually handled, so their classification is a tedious task (and therefore prone to errors) and habitually consumes a lot of time. The availability of automatic techniques to facilitate these sorting tasks would improve an important part of the digital documentation process. In addition, a correct classification of the available images allows better management and more efficient searches through specific terms, thus helping in the tasks of studying and interpreting the heritage asset in question. The main objective of this article is the application of techniques based on deep learning for the classification of images of architectural heritage, specifically through the use of convolutional neural networks. For this, the utility of training these networks from scratch or only fine tuning pre-trained networks is evaluated. All this has been applied to classifying elements of interest in images of buildings with architectural heritage value. As no datasets of this type, suitable for network training, have been located, a new dataset has been created and made available to the public. Promising results have been obtained in terms of accuracy and it is considered that the application of these techniques can contribute significantly to the digital documentation of architectural heritage.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: 3D Surveying and Cultural Heritage · Conservation Techniques and Studies · Infrastructure Maintenance and Monitoring
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
CARTIF Foundation, Parque Tecnológico de Boecillo, 47151 Valladolid, Spain
ITAP-DISA, University of Valladolid, Pl. Santa Cruz, 8, 47002 Valladolid, Spain
Funding
EU's H2020 Reflective frame-work programme · Ministry of Science and Innovation, Spain · Junta de Castilla y León, Spain
References (53)
Show source-provided references
1. Letellier, R., Schmid, W., and LeBlanc, F. (2007). Recording, Documentation, and Information Management for the Conservation of Heritage Places: Guiding Principles, Routledge.
2. Heritage Recording and 3D Modeling with Photogrammetry10.3390/rs3061104 ↗
3. (2017, September 25). CIPA Heritage Documentation. Available online: http://cipa.icomos.org/.
4. (2017, September 25). ICOMOS, International Council on Monuments & Sites. Available online: http://www.icomos.org/.
5. (2017, September 25). ISPRS, International Society of Photogrammetry and Remote Sensing. Available online: http://www.isprs.org/.
6. Digital Documentation in the Conservation of Cultural Heritage: Finding the Practical in best Practice10.5194/isprsarchives-XL-5-W2-85-2013 ↗
7. Documentation of cultural heritage; techniques, potentials, and constraints10.5194/isprsarchives-XL-5-W7-207-2015 ↗
8. López, F.J., Lerones, P.M., Llamas, J., Gómez-García-Bermejo, J., and Zalama, E. (2017). A framework for using point cloud data of heritage buildings towards geometry modeling in a BIM context: A case study on Santa Maria la Real de Mave Church. Int. J. Archit. Heritage, 11.10.1080/15583058.2017.1325541 ↗
9. A paradata documentation methodology for the Uncertainty Visualization in digital reconstruction of CH artifacts
10. Integrated Data Capturing Requirements for 3d Semantic Modelling of Cultural Heritage: The INCEPTION Protocol10.5194/isprs-archives-XLII-2-W3-251-2017 ↗
11. Image-based delineation and classification of built heritage masonry10.3390/rs6031863 ↗
12. Region-Based Convolutional Networks for Accurate Object Detection and Segmentation10.1109/TPAMI.2015.2437384 ↗
13. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully Convolutional Networks for Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
14. Cireşan, D., Meier, U., and Schmidhuber, J. (2012, January 16–21). Multi-column deep neural networks for image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248110 ↗
15. He, K., Zhang, X., Ren, S., and Sun, J. (arXiv, 2015). Deep residual learning for image recognition, arXiv.10.1109/CVPR.2016.90 ↗
16. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
17. Simonyan, K., and Zisserman, A. (arXiv, 2014). Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv.
18. Szegedy, C., Toshev, A., and Erhan, D. (2013, January 5–10). Deep neural networks for object detection. Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
19. Mnih, V., and Hinton, G. (July, January 27). Learning to Label Aerial Images from Noisy Data. Proceedings of the 29th International Conference on Machine Learning (ICML-12), Edinburgh, UK.
20. Gao, F., Huang, T., Wang, J., Sun, J., Hussain, A., and Yang, E. (2017). Dual-Branch Deep Convolution Neural Network for Polarimetric SAR Image Classification. Appl. Sci., 7.10.3390/app7050447 ↗
21. Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?10.1109/TMI.2016.2535302 ↗
22. Gao, Y., and Lee, H.J. (2016). Local Tiled Deep Networks for Recognition of Vehicle Make and Model. Sensors, 16.10.3390/s16020226 ↗
23. Li, C., Min, X., Sun, S., Lin, W., and Tang, Z. (2017). DeepGait: A Learning Deep Convolutional Representation for View-Invariant Gait Recognition Using Joint Bayesian. Appl. Sci., 7.10.3390/app7030210 ↗
24. Pedraza, A., Bueno, G., Deniz, O., Cristóbal, G., Blanco, S., and Borrego-Ramos, M. (2017). Automated Diatom Classification (Part B): A Deep Learning Approach. Appl. Sci., 7.10.3390/app7050460 ↗
25. Liu, L., Wang, H., and Wu, C. (arXiv, 2016). A machine learning method for the large-scale evaluation of urban visual environment, arXiv.10.1016/j.compenvurbsys.2017.06.003 ↗
26. Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., and McCool, C. (2016). DeepFruits: A Fruit Detection System Using Deep Neural Networks. Sensors, 16.10.3390/s16081222 ↗
27. Chu, W.-T., and Tsai, M.-H. (2012, January 5–8). Visual pattern discovery for architecture image classification and product image search. Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, Hong Kong, China.10.1145/2324796.2324831 ↗
28. Goel, A., Juneja, M., and Jawahar, C.V. (2012, January 16–19). Are buildings only instances?: Exploration in architectural style categories. Proceedings of the Eighth Indian Conference on Computer Vision, Graphics and Image Processing, Mumbai, India.10.1145/2425333.2425334 ↗
29. Mathias, M., Martinovic, A., Weissenberg, J., Haegler, S., and Van Gool, L. (2011, January 2–4). Automatic Architectural Style Recognition. Proceedings of the 4th ISPRS International Workshop 3D-ARCH 2011, Trento, Italy.
30. Shalunts, G., Haxhimusa, Y., and Sablatni, R. (2011). Architectural Style Classification of Building Facade Windows. Advances in Visual Computing 6939, Springer.10.1007/978-3-642-24031-7_28 ↗
31. Recognizing architecture styles by hierarchical sparse coding of blocklets10.1016/j.ins.2013.08.020 ↗
32. Architectural Style Classification Using Multinomial Latent Logistic Regression10.1007/978-3-319-10590-1_39 ↗
33. Applying Deep Learning Techniques to Cultural Heritage Images within the INCEPTION Project
34. Lu, Y. (arXiv, 2016). Food Image Recognition by Using Convolutional Neural Networks (CNNs), arXiv.
35. Yanai, K., and Kawano, Y. (July, January 29). Food image recognition using deep convolutional network with pre-training and fine-tuning. Proceedings of the IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Turin, Italy.
36. (2017, September 25). Datahub. Available online: https://datahub.io.
37. Liang, H., and Li, Q. (2016). Hyperspectral Imagery Classification Using Sparse Representations of Convolutional Neural Network Features. Remote Sens., 8.10.3390/rs8020099 ↗
38. Bengio, Y. (2011, January 2). Deep Learning of Representations for Unsupervised and Transfer Learning. Proceedings of the ICML Workshop on Unsupervised and Transfer Learning, Bellevue, WA, USA.
39. YFCC100m (2017, September 25). In: Yahoo Flickr Creative Commons 100 Million Dataset. Available online: http://www.yfcc100m.org/.
40. (2017, September 25). Getty Art & Architecture Thesaurus (AAT). Available online: http://www.getty.edu/research/tools/vocabularies/aat/about.html.
41. Deep learning10.1038/nature14539 ↗
42. (2017, September 25). ImageNet. Available online: http://www.image-net.org.
43. (2017, September 25). MIT Places. Available online: http://places.csail.mit.edu/.
44. Werbos, P. (September, January 31). Applications of advances in nonlinear sensitivity analysis. Proceedings of the 10th IFIP Conference, New York, NY, USA.
45. Learning internal representations by error propagation
46. Dundar, A., Jin, J., and Culurciello, E. (arXiv, 2015). Convolutional Clustering for Unsupervised Learning, arXiv.
47. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (arXiv, 2016). TensorFlow: Large-scale machine learning on heterogeneous systems, arXiv.
48. Bengio, Y. (arXiv, 2012). Practical recommendations for gradient-based training of deep architectures, arXiv.10.1007/978-3-642-35289-8_26 ↗
49. Bottou, L. (2012). Stochastic Gradient Descent Tricks. Neural Networks, Tricks of the Trade, Reloaded 7700, Springer.10.1007/978-3-642-35289-8_25 ↗
50. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
51. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (arXiv, 2015). Rethinking the inception architecture for computer vision, arXiv.10.1109/CVPR.2016.308 ↗
52. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (arXiv, 2016). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, arXiv.10.1609/aaai.v31i1.11231 ↗
53. Canziani, A., Paszke, A., and Culurciello, E. (arXiv, 2016). An Analysis of Deep Neural Network Models for Practical Applications, arXiv.