The Impacts of Open Data and eXplainable AI on Real Estate Price Predictions in Smart Cities
Source abstract
Abstract
In the rapidly evolving landscape of urban development, where smart cities increasingly rely on artificial intelligence (AI) solutions to address complex challenges, using AI to accurately predict real estate prices becomes a multifaceted and crucial task integral to urban planning and economic development. This paper delves into this endeavor, highlighting the transformative impact of specifically chosen contextual open data and recent advances in eXplainable AI (XAI) to improve the accuracy and transparency of real estate price predictions within smart cities. Focusing on Lisbon’s dynamic housing market from 2018 to 2021, we integrate diverse open data sources into an eXtreme Gradient Boosting (XGBoost) machine learning model optimized with the Optuna hyperparameter framework to enhance its predictive precision. Our initial model achieved a Mean Absolute Error (MAE) of EUR 51,733.88, which was significantly reduced by 8.24% upon incorporating open data features. This substantial improvement underscores open data’s potential to boost real estate price predictions. Additionally, we employed SHapley Additive exPlanations (SHAP) to address the transparency of our model. This approach clarifies the influence of each predictor on price estimates and fosters enhanced accountability and trust in AI-driven real estate analytics. The findings of this study emphasize the role of XAI and the value of open data in enhancing the transparency and efficacy of AI-driven urban development, explicitly demonstrating how they contribute to more accurate and insightful real estate analytics, thereby informing and improving policy decisions for the sustainable development of smart cities.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Explainable Artificial Intelligence (XAI) · Energy Load and Power Forecasting · Forecasting Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
NOVA Information Management School (NOVA IMS), Universidade Nova de Lisboa (UNL), Campus de Campolide, 1070-312 Lisboa, Portugal
Funding
FCT—Fundação para a Ciência e Tecnologia, I.P. (Portugal)
References (100+)
Show source-provided references
1. On big data, artificial intelligence and smart cities10.1016/j.cities.2019.01.032 ↗
2. The impacts of open data initiatives on smart cities: A framework for evaluation and monitoring10.1016/j.cities.2020.102860 ↗
3. Environmentally sustainable smart cities and their converging AI, IoT, and big data technologies and solutions: An integrated approach to an extensive literature review10.1186/s42162-023-00259-2 ↗
4. AI-Based on Machine Learning Methods for Urban Real Estate Prediction: A Systematic Survey10.1007/s11831-023-10010-5 ↗
5. Costa, C.J., and Aparicio, M. (2023). Applications of Data Science and Artificial Intelligence. Appl. Sci., 13.10.3390/app13159015 ↗
6. Yigitcanlar, T., Corchado, J.M., Mehmood, R., Li, R.Y.M., Mossberger, K., and Desouza, K. (2021). Responsible urban innovation with local government artificial intelligence (AI): A conceptual framework and research agenda. J. Open Innov. Technol. Mark. Complex., 7.10.3390/joitmc7010071 ↗
7. Koseki, S., Jameson, S., Farnadi, G., Rolnick, D., Régis, C., and Denis, J.-L. (2022). AI and Cities: Risks, Applications, and Governance, UN-Habitat.
8. Adoption of artificial intelligence in smart cities: A comprehensive review
9. Multi-head attention-based probabilistic CNN-BiLSTM for day-ahead wind speed forecasting10.1016/j.energy.2023.127865 ↗
10. Probabilistic framework with bayesian optimization for predicting typhoon-induced dynamic responses of a long-span bridge10.1061/(ASCE)ST.1943-541X.0002881 ↗
11. Linked open government data to predict and explain house prices: The case of Scottish statistics portal10.1016/j.bdr.2022.100355 ↗
12. Hurbean, L., Danaiata, D., Militaru, F., Dodea, A.-M., and Negovan, A.-M. (2021). Open data based machine learning applications in smart cities: A systematic literature review. Electronics, 10.10.3390/electronics10232997 ↗
13. Goodey, G., Hahnel, M., Zhou, Y., Jiang, L., Chandramouliswaran, I., Hafez, A., Paine, T., Gregurick, S., Simango, S., and Palma Peña, J.M. (2022). The State of Open Data 2022, Digital Science. Digital Science Report.
14. Davies, T., Walker, S.B., Rubinstein, M., and Perini, F. (2019). The State of Open Data: Histories and Horizons, African Minds.10.47622/9781928331957 ↗
15. Smart City Based on Open Data: A Survey10.1109/ACCESS.2023.3283436 ↗
16. Analysing Urban Growth Using Machine Learning and Open Data: An Artificial Neural Network Modelled Case Study of Five Greek Cities10.1016/j.scs.2022.104337 ↗
17. Smart city research advances in Southeast Europe10.1016/j.ijinfomgt.2020.102127 ↗
18. The economic and social impacts of smart cities: Multi-stakeholder pre-study results10.25019/71fq6q53 ↗
19. Housing Infrastructure as a Determinant of Quality of Life in Selected Polish Smart Cities10.3390/smartcities5030046 ↗
20. Gutman, S., and Rytova, E. (2019, January 24–25). Indicators for assessing the development of smart sustainable cities. Proceedings of the International Scientific Conference on Innovations in Digital Economy, Saint-Petersburg, Russia.10.1007/978-3-030-60080-8_4 ↗
21. Points of Interest and Housing Prices10.2478/remav-2023-0007 ↗
22. Nijskens, R., Lohuis, M., Hilbers, P., and Heeringa, W. (2019). Hot Property: The Housing Market in Major Cities, Springer Nature.10.1007/978-3-030-11674-3 ↗
23. Analyzing the effects of walkable environments on nearby commercial property values based on deep learning approaches10.1016/j.cities.2023.104628 ↗
24. Disentangling Housing Supply to Shift towards Smart Cities: Analysing Theoretical and Empirical Studies10.3390/smartcities5040076 ↗
25. Sustainable formation of urban development on the example of the primary real estate market in Krakow10.1051/e3sconf/20198600010 ↗
26. A housing supply absorption rate equation10.1007/s11146-020-09815-z ↗
27. House price forecasting with neural networks
28. Chollet, F. (2021). Deep Learning with Python, Simon and Schuster.
29. Uncovering the shape of neighborhoods: Harnessing data analytics for a smart governance of urban areas10.1016/j.jum.2022.05.005 ↗
30. Artificial intelligence algorithms to predict Italian real estate market prices10.1108/JPIF-08-2021-0073 ↗
31. Predicting property prices with machine learning algorithms10.1080/09599916.2020.1832558 ↗
32. Real estate price estimation in French cities using geocoding and machine learning10.1007/s10479-021-03932-5 ↗
33. Predicting property price index using artificial intelligence techniques Evidence from Hong Kong10.1108/IJHMA-11-2018-0095 ↗
34. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
35. Predictability of Belgian residential real estate rents using tree-based ML models and IML techniques10.1108/IJHMA-11-2022-0172 ↗
36. Extracting spatial effects from machine learning model using local interpretation method: An example of SHAP and XGBoost10.1016/j.compenvurbsys.2022.101845 ↗
37. Automated real estate valuation with machine learning models using property descriptions10.1016/j.eswa.2022.119147 ↗
38. Javed, A.R., Ahmed, W., Pandya, S., Maddikunta, P.K.R., Alazab, M., and Gadekallu, T.R. (2023). A survey of explainable artificial intelligence for smart cities. Electronics, 12.10.3390/electronics12041020 ↗
39. Disparities in affecting factors of housing price: A machine learning approach to the effects of housing status, public transit, and density factors on single-family housing price10.1016/j.cities.2023.104432 ↗
40. Molnar, C. (2022). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, Lulu.
41. Verhulst, S. (2023). Unlocking the Potential: The Call for an International Decade of Data, UNU-CPR.10.2139/ssrn.4647179 ↗
42. The Role of Explainable AI in the Research Field of AI Ethics
43. Royal Society (2019). Explainable AI: The Basics-Policy Briefing, Royal Society.
44. Exploring XAI techniques for enhancing model transparency and interpretability in real estate rent prediction: A comparative study10.1016/j.frl.2023.104306 ↗
45. Popelka, S., Zertuche, L., and Beroche, H. (2023). Urban AI Guide, Urban AI.
46. Explainable spatially explicit geospatial artificial intelligence in urban analytics
47. Lundberg, S.M., and Lee, S.-I. (2017, January 4–9). A unified approach to interpreting model predictions. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
48. Harold William, K., and Albert William, T. (1953). Contributions to the Theory of Games (AM-28), Volume II, Princeton University Press.
49. From local explanations to global understanding with explainable AI for trees10.1038/s42256-019-0138-9 ↗
50. The role of the state in the touristification of Lisbon10.1016/j.cities.2023.104275 ↗
51. Marques, T.S., Saraiva, M.M., Matos, F.L.d., Maia, C., Ribeiro, D., Ferreira, M., and Van Heerden, S. (2022). Property Investment and Housing Affordability in Lisbon and Porto, Publications Office of the European Union, Joint Research Centre (JRC).
52. Samadani, S., and Costa, C.J. (2021, January 23–26). Forecasting real estate prices in Portugal: A data science approach. Proceedings of the 2021 16th Iberian Conference on Information Systems and Technologies (CISTI), Chaves, Portugal.10.23919/CISTI52073.2021.9476447 ↗
53. Ahrend, R., Bétin, M., Caldas, M.P., Cournède, B., Ramirez, M.D., Pionnier, P.-A., Sanchez-Serra, D., Veneri, P., and Ziemann, V. (2023). Changes in the Geography of Housing Demand after the Onset of COVID-19, OECD.10.1787/79b957e5-en ↗
54. Using open data and open-source software to develop spatial indicators of urban design and transport features for achieving healthy and sustainable cities10.1016/S2214-109X(22)00072-9 ↗
55. Announcement of the First Metro Line and its Impact on Housing Prices in Bogotá
56. Neural network hyperparameter optimization for prediction of real estate prices in Helsinki10.7717/peerj-cs.444 ↗
57. Shen, H., Li, L., Zhu, H.H., and Li, F. (2022). A Pricing Model for Urban Rental Housing Based on Convolutional Neural Networks and Spatial Density: A Case Study of Wuhan, China. Isprs Int. J. Geo-Inf., 11.10.3390/ijgi11010053 ↗
58. Büchler, S., Niu, D., and Kinsella Thompson, A. (2021). Predicting Urban Growth with Machine Learning, MIT Center for Real Estate Research Paper.10.2139/ssrn.3784787 ↗
59. Bouwknegt, L., and Rouwendal, J. (2023). The Effect of Urban Trees on House Prices: Evidence from Cut-Down Trees in Amsterdam, Tinbergen Institute.
60. Improving performance of mass real estate valuation through application of the dataset optimization and Spatially Constrained Multivariate Clustering Analysis10.1016/j.landusepol.2022.106167 ↗
61. Walking accessibility and property prices10.1016/j.trd.2018.04.001 ↗
62. A multi-level modeling approach for predicting real-estate dynamics10.1108/IJHMA-02-2023-0024 ↗
63. The use of urban indicators in forecasting a real estate value with the use of deep neural network
64. Hacıevliyagil, N., Drachal, K., and Eksi, I.H. (2022). Predicting House Prices Using DMA Method: Evidence from Turkey. Economies, 10.10.3390/economies10030064 ↗
65. Guliker, E., Folmer, E., and van Sinderen, M. (2022). Spatial determinants of real estate appraisals in the Netherlands: A machine learning approach. ISPRS Int. J. Geo-Inf., 11.10.3390/ijgi11020125 ↗
66. Barcelos, M., Barcelos, A., Bernardini, F., and Silva, G.V. (June, January 30). Analyzing the use of economic and financial indicators in smart cities context. Proceedings of the 19th Annual International Conference on Digital Government Research: Governance in the Data Age, Delft, The Netherlands.
67. Wu, H.-N., Yin, L., Zhou, T., and Ye, S. (2017, January 23–26). City smart-growth evaluation system. Proceedings of the 2017 IEEE International Conference on Smart Grid and Smart Cities (ICSGSC), Singapore.10.1109/ICSGSC.2017.8038593 ↗
68. García, S., Luengo, J., and Herrera, F. (2015). Data Preprocessing in Data Mining, Springer.10.1007/978-3-319-10247-4 ↗
69. Iglewicz, B., and Hoaglin, D.C. (1993). Volume 16: How to Detect and Handle Outliers, Quality Press.
70. House price forecasting using machine learning methods
71. Chen, T., and Guestrin, C. (2016, January 13–17). Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939785 ↗
72. A Comparative Study of Urban House Price Prediction using Machine Learning Algorithms10.1051/e3sconf/202341803001 ↗
73. Jha, S.B., Babiceanu, R.F., Pandey, V., and Jha, R.K. (2020). Housing market prediction problem using different machine learning algorithms: A case study. arXiv.
74. Greedy function approximation: A gradient boosting machine10.1214/aos/1013203451 ↗
75. Comparative Performance Evaluation of State-of-the-Art Hyperparameter Optimization Frameworks10.5370/KIEE.2023.72.5.607 ↗
76. Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019, January 4–8). Optuna: A next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.10.1145/3292500.3330701 ↗
77. Modified Prophet+ Optuna Prediction Method for Sales Estimations
78. Lai, J.-P., Lin, Y.-L., Lin, H.-C., Shih, C.-Y., Wang, Y.-P., and Pai, P.-F. (2023). Tree-Based Machine Learning Models with Optuna in Predicting Impedance Values for Circuit Analysis. Micromachines, 14.10.3390/mi14020265 ↗
79. Joy, J., and Selvan, M.P. (2022, January 23–25). A comprehensive study on the performance of different Multi-class Classification Algorithms and Hyperparameter Tuning Techniques using Optuna. Proceedings of the 2022 International Conference on Computing, Communication, Security and Intelligent Systems (IC3SIS), Kochi, India.10.1109/IC3SIS54991.2022.9885695 ↗
80. Elshewey, A.M. (2023). hyOPTGB: An Efficient OPTUNA Hyperparameter Optimization Framework for Hepatitis C Virus (HCV) Disease Prediction in Egypt. Res. Sq.10.21203/rs.3.rs-2768795/v1 ↗
81. Agrawal, T. (2021). Machine Learning: Make Your Machine Learning and Deep Learning Models More Efficient, Springer.
82. Eimer, T., Lindauer, M., and Raileanu, R. (2023). Hyperparameters in Reinforcement Learning and How To Tune Them. arXiv.
83. Shekhar, S., Bansode, A., and Salim, A. (2021, January 8–10). A comparative study of hyper-parameter optimization tools. Proceedings of the 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Brisbane, Australia.10.1109/CSDE53843.2021.9718485 ↗
84. Metrics for evaluating the performance of machine learning based automated valuation models10.1080/09599916.2020.1858937 ↗
85. De Castro Neto, M., and De Melo Cartaxo, T. (2021). Algorithmic Cities: A Dystopic or Utopic Future?, Springer International Publishing.10.1007/978-3-030-56926-6_6 ↗
86. Carvalho, D.V., Pereira, E.M., and Cardoso, J.S. (2019). Machine learning interpretability: A survey on methods and metrics. Electronics, 8.10.3390/electronics8080832 ↗
87. Van Lent, M., Fisher, W., and Mancuso, M. (2004, January 25–29). An explainable artificial intelligence system for small-unit tactical behavior. Proceedings of the National Conference on Artificial Intelligence, San Jose, CA, USA.
88. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why should i trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
89. The measurement and early warning of daily financial stability index based on XGBoost and SHAP: Evidence from China10.1016/j.eswa.2023.120375 ↗
90. Silva-Aravena, F., Núñez Delafuente, H., Gutiérrez-Bahamondes, J.H., and Morales, J. (2023). A hybrid algorithm of ML and XAI to prevent breast cancer: A strategy to support decision making. Cancers, 15.10.3390/cancers15092443 ↗
91. Use of machine learning in understanding transport dynamics of land use and public transportation in a developing city10.1016/j.cities.2023.104587 ↗
92. Kansal, M., Singh, P., Shukla, S., and Srivastava, S. (2023, January 11–12). A Comparative Study of Machine Learning Models for House Price Prediction and Analysis in Smart Cities. Proceedings of the International Conference on Electronic Governance with Emerging Technologies, Poznan, Poland.10.1007/978-3-031-43940-7_14 ↗
93. An explainable model for the mass appraisal of residences: The application of tree-based Machine Learning algorithms and interpretation of value determinants10.1016/j.habitatint.2022.102660 ↗
94. Xu, K., and Nguyen, H. (2022). Predicting housing prices and analyzing real estate market in the Chicago suburbs using Machine Learning. arXiv.10.47611/jsrhs.v11i3.3459 ↗
95. Machine learning with explainability or spatial hedonics tools? An analysis of the asking prices in the housing market in Alicante, Spain10.1016/j.eswa.2021.114590 ↗
96. Improving property valuation accuracy: A comparison of hedonic pricing model and artificial neural network
97. Morano, P., Tajani, F., Di Liddo, F., and Darò, M. (2021). Economic evaluation of the indoor environmental quality of buildings: The noise pollution effects on housing prices in the city of Bari (Italy). Buildings, 11.10.3390/buildings11050213 ↗
98. Explainable housing price prediction with determinant analysis10.1108/IJHMA-02-2022-0025 ↗
99. Zhang, X., and Yang, E. (Int. J. Hous. Mark. Anal., 2023). Observation of relationship between housing value and the number of building permits in the United States using time series method, Int. J. Hous. Mark. Anal., ahead-of-print.10.1108/IJHMA-06-2023-0085 ↗
100. Deppner, J., von Ahlefeldt-Dehn, B., Beracha, E., and Schaefers, W. (2023). Boosting the Accuracy of Commercial Real Estate Appraisals: An Interpretable Machine Learning Approach. J. Real Estate Financ. Econ., 1–38.10.1007/s11146-023-09944-1 ↗