Between Truth and Hallucinations: Evaluation of the Performance of Large Language Model-Based AI Plugins in Website Quality Analysis
Source abstract
Abstract
Although large language models (LLMs) like the Generative Pre-trained Transformer (GPT) are growing increasingly popular, much remains to learn about their potential for website quality auditing. The article evaluates the performance of LLM AI plugins (GPT models) in website and web application auditing. The author built and tested two original ChatGPT-4o Plus (OpenAI) plugins: Website Quality Auditor (WQA) and WebGIS Quality Auditor (WgisQA). Their performance was cautiously and carefully analysed and compared to traditional auditing tools. The results demonstrated the limitations of the AI plugins, including their propensity for false outcomes. The general conclusion is that using AI tools without considering their characteristics may lead to the propagation of AI hallucinations in audit reports. The study fills in the research gap with the results on the capabilities and limitations of AI plugins in the context of auditing. It also suggests further directions for improvement.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Sentiment Analysis and Opinion Mining
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Land Management and Landscape Architecture, Faculty of Environmental Engineering and Land Surveying, University of Agriculture in Krakow, Balicka 253c, 30-198 Krakow, Poland
Funding
Minister of Science
References (62)
Show source-provided references
1. Szabó, Z., and Bilicki, V. (2023). A New Approach to Web Application Security: Utilizing GPT Language Models for Source Code Inspection. Future Internet, 15.10.3390/fi15100326 ↗
2. López-Gil, J.-M., and Pereira, J. (2024). Turning Manual Web Accessibility Success Criteria into Automatic: An LLM-Based Approach. Univ. Access Inf. Soc.10.1007/s10209-024-01108-z ↗
3. GPT (Generative Pre-Trained Transformer)—A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions10.1109/ACCESS.2024.3389497 ↗
4. Ma, P., Ding, R., Wang, S., Han, S., and Zhang, D. (2023, January 6–10). InsightPilot: An LLM-Empowered Automated Data Exploration System. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Singapore.10.18653/v1/2023.emnlp-demo.31 ↗
5. Leveraging Large Language Models for End-User Website Generation10.1007/978-3-031-34433-6_4 ↗
6. Othman, A., Dhouib, A., and Nasser Al Jabor, A. (2023, January 5–7). Fostering Websites Accessibility: A Case Study on the Use of the Large Language Models ChatGPT for Automatic Remediation. Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, Corfu Greece.10.1145/3594806.3596542 ↗
7. Artificial Hallucinations in ChatGPT: Implications in Scientific Writing
8. ChatGPT: Jack of All Trades, Master of None10.1016/j.inffus.2023.101861 ↗
9. Credibility and Trust of Information in Online Environments: The Use of Cognitive Heuristics10.1016/j.pragma.2013.07.012 ↗
10. Large Language Models for Search Engine Optimization in E-Commerce10.1007/978-3-031-56700-1_27 ↗
11. Predicting the Usability of Mobile Applications Using AI Tools: The Rise of Large User Interface Models, Opportunities, and Challenges10.1016/j.procs.2024.06.076 ↗
12. McGill, T., Bamgboye, O., Liu, X., and Kalutharage, C.S. (2023, January 26–30). Towards Improving Accessibility of Web Auditing with Google Lighthouse. Proceedings of the 2023 IEEE 47th Annual Computers, Software and Applications Conference (COMPSAC), Torino, Italy.10.1109/COMPSAC57700.2023.00246 ↗
13. Xu, R., and Li, G. (2024). A Comparative Study of Offline Models and Online LLMs in Fake News Detection. arXiv.
14. Ruan, J., Gao, J., Xie, M., Xiang, S., Yu, Z., Liu, T., Fu, Y., and Qu, X. (November, January 28). GIST: Improving Parameter Efficient Fine-Tuning via Knowledge Interaction. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, VIC, Australia.
15. Local Government Website Usability on Mobile Devices: Test Results and Recommendations
16. Lozić, E., and Štular, B. (2023). Fluent but Not Factual: A Comparative Analysis of ChatGPT and Other AI Chatbots’ Proficiency and Originality in Scientific Writing for Humanities. Future Internet, 15.10.3390/fi15100336 ↗
17. Kulkarni, A., Shivananda, A., Kulkarni, A., and Gudivada, D. (2023). The ChatGPT Architecture: An In-Depth Exploration of OpenAI’s Conversational Language Model. Applied Generative AI for Beginners, Apress.10.1007/978-1-4842-9994-4 ↗
18. War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education
19. Impact of Software as a Service (SaaS) on Software Acquisition Process10.1108/JBIM-12-2018-0382 ↗
20. Revolutionizing Language Processing in Libraries with SheetGPT: An Integration of Google Sheet and ChatGPT Plugin10.1108/LHTN-03-2023-0051 ↗
21. Jamdade, M., and Liu, Y. (2024, January 18–20). A Pilot Study on Secure Code Generation with ChatGPT for Web Applications. Proceedings of the 2024 ACM Southeast Conference on ZZZ, Marietta, GA, USA.10.1145/3603287.3651194 ↗
22. Martinez-Caro, J.-M., Aledo-Hernandez, A.-J., Guillen-Perez, A., Sanchez-Iborra, R., and Cano, M.-D. (2018). A Comparative Study of Web Content Management Systems. Information, 9.10.3390/info9020027 ↗
23. Visual Augmentation of Source Code Editors: A Systematic Mapping Study10.1016/j.jvlc.2018.10.001 ↗
24. Implementing Geographic Information System (GIS) Tools: Algorithm and Quantum Geographic Information System (QGIS) Plugin for Automatic Detection of Riverbed Forms10.1016/j.mex.2024.103041 ↗
25. Frazão, T., and Duarte, C. (2020, January 20–21). Comparing Accessibility Evaluation Plug-Ins. Proceedings of the 17th International Web for All Conference, Taipei, Taiwan.10.1145/3371300.3383346 ↗
26. FSLAM: A QGIS Plugin for Fast Regional Susceptibility Assessment of Rainfall-Induced Landslides10.1016/j.envsoft.2022.105354 ↗
27. Leveraging ChatGPT and Other Generative Artificial Intelligence (AI)-Based Applications in the Hospitality and Tourism Industry: Practices, Challenges and Research Agenda10.1108/IJCHM-05-2023-0686 ↗
28. Accessibility and Performance Evaluation of Healthcare and E-Learning Sites in India: A Comparative Study Using TAW and GTMetrix10.1007/978-981-19-2719-5_16 ↗
29. Shenoy, A., and Prabhu, A. (2016). SEO Hub: Utilities and Toolsets. Introducing SEO, Apress.10.1007/978-1-4842-1854-9 ↗
30. Giannakoulopoulos, A., Konstantinou, N., Koutsompolis, D., Pergantis, M., and Varlamis, I. (2019). Academic Excellence, Website Quality, SEO Performance: Is There a Correlation?. Future Internet, 11.10.3390/fi11110242 ↗
31. Game Changers: A Generative AI Prompt Protocol to Enhance Human-AI Knowledge Co-Construction10.1016/j.bushor.2024.04.008 ↗
32. Artificial Intelligence Prompt Engineering as a New Digital Competence: Analysis of Generative AI Technologies Such as ChatGPT
33. Perminov, N., and Bakaev, M. (2019, January 21–27). Assessing Quality of Web User Interfaces Using Domain Ontology. Proceedings of the 2019 International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON), Novosibirsk, Russia.10.1109/SIBIRCON48586.2019.8958023 ↗
34. Król, K., and Sroka, W. (2023). Internet in the Middle of Nowhere: Performance of Geoportals in Rural Areas According to Core Web Vitals. ISPRS Int. J. Geo-Inf., 12.10.3390/ijgi12120484 ↗
35. Aesthetic Assessment of Website Design Based on Multimodal Fusion10.1016/j.future.2020.12.014 ↗
36. Evaluating Websites for Older Adults: Adherence to ‘Senior-Friendly’ Guidelines and End-User Performance10.1080/01449290600802031 ↗
37. Knowledge Injection to Counter Large Language Model (LLM) Hallucination
38. ChatGPT: These Are Not Hallucinations–They’re Fabrications and Falsifications10.1038/s41537-023-00379-4 ↗
39. Harnessing the Power of LLMs for Service Quality Assessment From User-Generated Content10.1109/ACCESS.2024.3429290 ↗
40. Delnevo, G., Andruccioli, M., and Mirri, S. (2024, January 6–9). On the Interaction with Large Language Models for Web Accessibility: Implications and Challenges. Proceedings of the 2024 IEEE 21st Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA.10.1109/CCNC51664.2024.10454680 ↗
41. Król, K., Zdonek, D., and Sroka, W. (2024). Functionality Assessment Checklist for Evaluating Geoportals Useful in Planning Sustainable Tourism. Sustainability, 16.10.3390/su16125242 ↗
42. ChatGPT and Search Engine Optimisation: The Future Is Here
43. Zermeño, N., Zuheros, C., Calache, L.D.D.R., Herrera, F., and Montes Soldado, R. (2024). An Online A/B Testing Decision Support System for Web Usability Assessment Based on a Linguistic Decision-Making Model.10.2139/ssrn.4726022 ↗
44. Detecting Hallucinations in Large Language Models Using Semantic Entropy10.1038/s41586-024-07421-0 ↗
45. Kumar, A., Kumar, A., Hashmi, H., and Khan, S.A. (2021, January 10–11). WordPress: A Multi-Functional Content Management System. Proceedings of the 2021 10th International Conference on System Modeling & Advancement in Research Trends (SMART), Moradabad, India.10.1109/SMART52563.2021.9675311 ↗
46. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making
47. Can You Trust the Black Box? The Effect of Personality Traits on Trust in AI-Enabled User Interfaces
48. Faizin, M.A., Nevin, M., and Yuhana, U.L. (2024, January 28–29). Indonesia E-Government Website Performance and Accessibility Evaluation Using Automated Tool Lighthouse. Proceedings of the 2024 2nd International Conference on Software Engineering and Information Technology (ICoSEIT), Bandung, Indonesia.10.1109/ICoSEIT60086.2024.10497521 ↗
49. An Empirical Performance Evaluation of Universities Website
50. Website Accessibility in the Tourism Industry: An Analysis of Official National Tourism Organization Websites around the World10.1080/09638288.2017.1362709 ↗
51. Król, K., and Zdonek, D. (2020). Local Government Website Accessibility—Evidence from Poland. Adm. Sci., 10.10.3390/admsci10020022 ↗
52. Usability and Accessibility Assessment of Saudi Arabia Mobile E-Government Websites10.1109/ACCESS.2021.3068917 ↗
53. Individual Pension Products Offered by Banks in Poland—A Multidimensional Comparative Analysis
54. Readability and Clarity of Individual Pension Product Contracts10.1007/s11196-023-09997-8 ↗
55. A Webometric Study of Selected International Library Association Websites An Evaluative Study10.14429/djlit.42.3.17772 ↗
56. The Accessibility of University Web Sites: The Case of Turkish Universities10.1007/s10209-010-0190-z ↗
57. A Usability Analysis of Municipal Government Website Home Pages in Alabama10.1016/j.giq.2011.12.010 ↗
58. Link-Rot in Web-Sourced Multimedia Datasets
59. Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience10.1080/10447318.2015.1064648 ↗
60. Recent Advances in Automatic Black-Box Testing10.1016/bs.adcom.2015.04.002 ↗
61. Specification Matching of Software Components10.1145/261640.261641 ↗
62. Altulaihan, E.A., Alismail, A., and Frikha, M. (2023). A Survey on Web Application Penetration Testing. Electronics, 12.10.3390/electronics12051229 ↗