Preserving architectural heritage in urban renewal: a stable diffusion model framework for automated historical facade generation
Source abstract
Abstract
Preserving the architectural heritage of traditional historical districts is a crucial aspect of urban renewal. Traditional design workflows are time-consuming and subjective. Current data-driven design methods generate specific style images but are labor-intensive. Recent research highlights Stable Diffusion models’ potential in generating high-fidelity images based on prompts. However, research applying these models to historical districts is scarce, with challenges in creating effective prompts and training parameters. This study proposes a framework combining Stable Diffusion models with expert system-based techniques to generate architectural facades from professional prompts. We constructed a dataset of traditional arcade facades, trained Low-Rank Adaptation (LoRA) models, and integrated ControlNet models to enhance controllability. Experimental results showed our models excelled in precision, realism, and diversity. Both qualitative and quantitative evaluations, along with practical application tests, confirmed our approach aids designers and prompts engineers, contributing to the preservation of architectural heritage and the renewal of urban historical districts.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Conservation Techniques and Studies · Building Energy and Comfort Optimization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (60)
Show source-provided references
1. Adamus-Matuszyńska, A., Michnik, J. & Polok, G. A systemic approach to city image building. The case of Katowice city. Sustainability 11, 4470 (2019).10.3390/su11164470 ↗
2. Bulut, Y. & Atabeyoğlu, Ö Fountains as urban furniture in historical urban structure and usage culture: Erzurum city case. Build. Environ. 42, 2432–2438 (2007).10.1016/j.buildenv.2006.08.022 ↗
3. Ali, A. K. & Lee, O. J. Facade style mixing using artificial intelligence for urban infill. Architecture 3, 258–269 (2023).10.3390/architecture3020015 ↗
4. Sun, C., Zhou, Y. & Han, Y. Automatic generation of architecture facade for historical urban renovation using generative adversarial network. Build. Environ. 212, 108781 (2022).10.1016/j.buildenv.2022.108781 ↗
5. Sagˇsöz, A., Tuluk, Öİ & Özgen, S. Influences of different ages and cultures on each other from architectural point of view: examination of historical buildings in Trabzon/Turkiye. Build. Environ. 41, 45–59 (2006).10.1016/j.buildenv.2005.01.005 ↗
6. Abediniangerabi, B., Shahandashti, S. M. & Makhmalbaf, A. A data-driven framework for energy-conscious design of building facade systems. J. Build. Eng. 29, 101172 (2020).10.1016/j.jobe.2020.101172 ↗
7. Montali, J., Sauchelli, M., Jin, Q. & Overend, M. Knowledge-rich optimisation of prefabricated façades to support conceptual design. Autom. Constr. 97, 192–204 (2019).10.1016/j.autcon.2018.11.002 ↗
8. Chatzikonstantinou, I. & Sariyildiz, I. S. Addressing design preferences via auto-associative connectionist models: application in sustainable architectural Façade design. Autom. Constr. 83, 108–120 (2017).10.1016/j.autcon.2017.08.007 ↗
9. Piroozfar, P., Farr, E. R. P., Hvam, L., Robinson, D. & Shafiee, S. Configuration platform for customisation of design, manufacturing and assembly processes of building façade systems: a building information modelling perspective. Autom. Constr. 106, 102914 (2019).10.1016/j.autcon.2019.102914 ↗
10. Ma, K., Wang, B., Li, Y. & Zhang, J. Image retrieval for local architectural heritage recommendation based on deep hashing. Buildings 12, 809 (2022).10.3390/buildings12060809 ↗
11. Tang, P., Wang, X. & Shi, X. Generative design method of the facade of traditional architecture and settlement based on knowledge discovery and digital generation: a case study of Gunanjie Street in China. Int. J. Archit. Herit. 13, 679–690 (2019).10.1080/15583058.2018.1463415 ↗
12. Yüksel, N., Börklü, H. R., Sezer, H. K. & Canyurt, O. E. Review of artificial intelligence applications in engineering design perspective. Eng. Appl. Artif. Intell. 118, 105697 (2023).10.1016/j.engappai.2022.105697 ↗
13. Takhmasib, M., Lee, H. J. & Yi, H. Machine-learned kinetic Façade: construction and artificial intelligence enabled predictive control for visual comfort. Autom. Constr. 156, 105093 (2023).10.1016/j.autcon.2023.105093 ↗
14. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. High-resolution image synthesis with latent diffusion models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10684–10695 (2022).10.1109/CVPR52688.2022.01042 ↗
15. Hu, E. J. et al. LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 1(2), 3 (2022).
16. Zhang, L. & Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proc. IEEE/CVF International Conference on Computer Vision. 3836–3847 (2023).10.1109/ICCV51070.2023.00355 ↗
17. Sanchez, T. Examining the text-to-image community of practice: why and how do people prompt generative AIs? In Proc. 15th Conference on Creativity and Cognition 43–61 (Association for Computing Machinery, New York, NY, USA, 2023).10.1145/3591196.3593051 ↗
18. Liu, V. & Chilton, L. B. Design guidelines for prompt engineering text-to-image generative models. In Proc. 2022 CHI Conference on Human Factors in Computing Systems 1–23 (Association for Computing Machinery, New York, NY, USA, 2022).10.1145/3491102.3501825 ↗
19. Zhang, J., Xiang, R., Kuang, Z., Wang, B. & Li, Y. ArchGPT: harnessing large language models for supporting renovation and conservation of traditional architectural heritage. Herit. Sci. 12, 220 (2024).10.1186/s40494-024-01334-x ↗
20. Jo, H., Lee, J.-K., Lee, Y.-C. & Choo, S. Generative artificial intelligence and building design: early photorealistic render visualization of façades using local identity-trained models. J. Comput. Des. Eng. 11, 85–105 (2024).
21. Zheng, H. A diffusion-based machine learning method for 3D architectural form-finding. Front. Archit. Res. https://doi.org/10.1016/j.foar.2024.12.002 (in the press).10.1016/j.foar.2024.12.002 ↗
22. Chen, J. et al. Creative interior design matching the indoor structure generated through diffusion model with an improved control network. Front. Archit. Res. 14, 614–629 (2025).10.1016/j.foar.2024.08.003 ↗
23. Cao, Y., Abdul Aziz, A. & Mohd Arshard, W. N. R. Stable diffusion in architectural design: closing doors or opening new horizons? Int. J. Archit. Comput. 14780771241270257 (2024).10.1177/14780771241270257 ↗
24. Pulles, K. et al. Emerging strategies for regeneration of historic urban sites: a systematic literature review. City, Cult. Soc. 35, 100539 (2023).10.1016/j.ccs.2023.100539 ↗
25. Vardopoulos, I. Industrial building adaptive reuse for museum. Factors affecting visitors’ perceptions of the sustainable urban development potential. Build. Environ. 222, 109391 (2022).10.1016/j.buildenv.2022.109391 ↗
26. Soave, A. & Hawizy, B. Rebuilding Tajeel: strategies to reverse the deterioration of cultural heritage and loss of identity of the historic quarters of Erbil, Kurdistan, Iraq. In Urban Heritage Along the Silk Roads: A Contemporary Reading of Urban Transformation of Historic Cities in the Middle East and Beyond (eds Arefian, F. F. & Moeini, S. H. I.) 75–91 (Springer International Publishing, Cham, 2020).10.1007/978-3-030-22762-3_6 ↗
27. Shan, L. & Zhang, L. Application of intelligent technology in facade style recognition of Harbin modern architecture. Sustainability 14, 7073 (2022).10.3390/su14127073 ↗
28. Xia, B., Li, X., Shi, H., Chen, S. & Chen, J. Style classification and prediction of residential buildings based on machine learning. J. Asian Archit. Build. Eng. 19, 714–730 (2020).10.1080/13467581.2020.1779728 ↗
29. Foster, G. Circular economy strategies for adaptive reuse of cultural heritage buildings to reduce environmental impacts. Resour. Conserv. Recycl. 152, 104507 (2020).10.1016/j.resconrec.2019.104507 ↗
30. Silveira da Costa, V., Montagna da Silveira, A. & da Silva Torres, A. Evaluation of degradation state of historic building facades through qualitative and quantitative indicators: case study in Pelotas, Brazil. Int. J. Archit. Herit. 16, 1642–1665 (2022).10.1080/15583058.2021.1901161 ↗
31. Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. 1125–1134 (2017).10.1109/CVPR.2017.632 ↗
32. Larrain, T. V., Valencia, A. & Yuan, P. F. Spatial findings on Chilean architecture StyleGAN AI graphics. In Proc. 26th International Conference of the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA). 251–260 (Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong, 2021).10.52842/conf.caadria.2021.1.251 ↗
33. Bachl, M. & Ferreira, D. C. City-GAN: learning architectural styles using a custom Conditional GAN architecture. Preprint at https://arxiv.org/abs/1907.05280 (2019).
34. Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV). 2242–2251 (2017).10.1109/ICCV.2017.244 ↗
35. Zhang, J., Fukuda, T., Yabuki, N. & Li, Y. Synthesizing style-similar residential facade from semantic labeling according to the user-provided example. In HUMAN-CENTRIC, In Proceedings of the 28th International Conference of the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Ahmedebad, India, 18 March 2023, vol. 1, 139–148 (Hong Kong, China, 2023).10.52842/conf.caadria.2023.1.139 ↗
36. Zhang, J., Fukuda, T. & Yabuki, N. Automatic object removal with obstructed façades completion using semantic segmentation and generative adversarial inpainting. IEEE Access 9, 117486–117495 (2021).10.1109/ACCESS.2021.3106124 ↗
37. Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 6840–6851 (Curran Associates, Inc., 2020).
38. Dhariwal, P. & Nichol, A. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems 8780–8794 (Curran Associates, Inc., 2021).
39. Radford, A. et al. Learning transferable visual models from natural language supervision. In Proc. 38th International Conference on Machine Learning 8748–8763 (PMLR, 2021).
40. Gu, S. et al. Vector quantized diffusion model for text-to-image synthesis. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10696–10706 (2022).10.1109/CVPR52688.2022.01043 ↗
41. Kawar, B. et al. Imagic: text-based real image editing with diffusion models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6007–6017 (2023).10.1109/CVPR52729.2023.00582 ↗
42. Avrahami, O., Lischinski, D. & Fried, O. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18208–18218 (2022).10.1109/CVPR52688.2022.01767 ↗
43. Borji, A. Generated faces in the wild: quantitative comparison of stable diffusion, midjourney and DALL-E 2. Preprint at https://arxiv.org/abs/2210.00586 (2022).
44. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C. & Chen, M. Hierarchical text-conditional image generation with CLIP latents. Preprint at https://arxiv.org/abs/2204.06125 (2022).
45. Saharia, C. et al. Palette: image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings 1–10 (Association for Computing Machinery, New York, NY, USA, 2022).10.1145/3528233.3530757 ↗
46. Chen, J., Shao, Z. & Hu, B. Generating interior design from text: a new diffusion model-based method for efficient creative design. Buildings 13, 1861 (2023).10.3390/buildings13071861 ↗
47. Ruiz, N. et al. DreamBooth: fine tuning text-to-image diffusion models for subject-driven generation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22500-22510 (2023).10.1109/CVPR52729.2023.02155 ↗
48. Zhang, H. et al. Layered analysis of irregular facades via symmetry maximization. ACM Trans. Graph. 32, 1–13 (2013).
49. Fan, L., Musialski, P., Liu, L. & Wonka, P. Structure completion for facade layouts. ACM Trans. Graph. 33, 210:1–210:11 (2014).10.1145/2661229.2661265 ↗
50. Agirbas, A. Façade form-finding with swarm intelligence. Autom. Constr. 99, 140–151 (2019).10.1016/j.autcon.2018.12.003 ↗
51. Kuang, Z., Zhang, J., Huang, Y. & Li, Y. Advancing urban renewal: an automated approach to generating historical arcade facades with stable diffusion models. In Proc. 43rd ACADIA Conference - Volume II: Proceedings Book One. 616–625 (Denver, CO, USA, 2023).
52. Tan, X. & Altrock, U. Struggling for an adaptive strategy? Discourse analysis of urban regeneration processes – A case study of Enning Road in Guangzhou City. Habitat Int. 56, 245–257 (2016).10.1016/j.habitatint.2016.06.006 ↗
53. Huang, X. Reinterpretation of Arcade Typology in a Contemporary Way. Master’s thesis, School of Arts, Design and Architecture, Aalto University, Espoo https://aaltodoc.aalto.fi/items/26c9bac8-a091-4d6a-8c4e-97df926afad8 (2017).
54. Kichang, K. DeepDanbooru: AI based multi-label girl image classification system, implemented by using TensorFlow. GitHub https://github.com/KichangKim/DeepDanbooru (2024).
55. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. & Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems (Curran Associates, Inc., 2017).
56. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. 2818–2826 (2016).10.1109/CVPR.2016.308 ↗
57. DeVries, T., Romero, A., Pineda, L., Taylor, G. W. & Drozdzal, M. On the evaluation of conditional GANs. Preprint at https://arxiv.org/abs/1907.08175 (2019).
58. Hu, Y. et al. TIFA: accurate and interpretable text-to-image faithfulness evaluation with question answering. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) 20349–20360 (IEEE, Paris, France, 2023).10.1109/ICCV51070.2023.01866 ↗
59. Betker, J. et al. Improving image generation with better captions. Comput. Sci. 2, 8 https://cdn.openai.com/papers/dall-e-3 pdf (2023).
60. Podell, D. et al. SDXL: improving latent diffusion models for high-resolution image synthesis. Preprint at https://arxiv.org/abs/2307.01952 (ICLR, 2024).