Hybrid Usability Evaluation of an Automotive REM Tool: Human and LLM-Based Heuristic Assessment of IBM Doors Next
Source abstract
Abstract
Requirements Engineering and Management (REM) tools play a significant role in ensuring project compliance and efficiency. Automotive engineering must comply with regulatory standards, requiring detailed documentation, rigorous testing, and solid traceability. Despite their importance, REM tools are underexplored from the usability and user experience perspective (UX), even though poor usability can hinder development workflows across stakeholder teams. This study presents a case study of heuristic usability evaluation of IBM DOORS Next Generation, conducted with expert evaluators, using Nielsen’s 10 Usability Heuristics as an evaluation framework. The identified issues were analyzed in terms of impacted heuristics and severity ratings. Additionally, we underwent a Large Language Model (LLM)-based heuristic evaluation, using ChatGPT-5, prompted with the same heuristic set and static screenshots. The LLM detected several issues overlapping with human findings (32%), as well as new ones (23%); therefore, 55% of its outputs are considered valid and 45% are unconfirmed. This highlights both the potential and limitations of AI-driven usability assessment. Overall, the findings underscore the usability challenges of REM tools and suggest that LLMs may serve as complementary evaluators, accelerating early-stage heuristic inspections in safety-critical engineering environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Techniques and Practices · Human-Automation Interaction and Safety · Usability and User Interface Design
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Communications, Politehnica University of Timișoara, 300006 Timișoara, Romania
AUMOVIO Timisoara, 300704 Timișoara, Romania
References (66)
Show source-provided references
1. Evolution of IoT-enabled connectivity and applications in automotive industry: A review
2. Regnell, B., and Damian, D. (2012). High-Level Requirements Management and Complexity Costs in Automotive Development Projects: A Problem Statement. Requirements Engineering: Foundation for Software Quality, Springer.
3. Agile and traditional requirements engineering: A survey
4. (2024, October 01). Overview of Rational DOORS Next Generation—IBM Documentation. Available online: https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/doors/9.7.0?topic=doors-overview-rational-next-generation.
5. (2025, September 03). 2025 Siemens Product Lifecycle Management Software Inc. Requirements Management, Requirements Gathering, Requirements Management Tools—Polarion REQUIREMENTS. Available online: https://polarion.plm.automation.siemens.com/products/polarion-requirements.
6. (2025, May 13). Jama Connect. Available online: https://www.jamasoftware.com/platform/jama-connect/.
7. Types and Benefits of Requirements Management Tools
8. (2018). Road Vehicles—Functional Safety (Standard No. ISO 26262-4:2018). Available online: https://www.iso.org/standard/68386.html.
9. (2025, May 17). Automotive SPICE Process Assessment/Reference Model. Available online: https://vda-qmc.de/en/automotive-spice/.
10. Lami, G., and Falcini, F. (2014, January 3–6). Automotive SPICE Assessments in Safety-Critical Contexts: An Experience Report. Proceedings of the 2014 IEEE International Symposium on Software Reliability Engineering Workshops, Naples, Italy.10.1109/ISSREW.2014.81 ↗
11. Khan, J. (2017, January 21–22). ISO 26262 system level functional safety validation for battery management systems in automobiles. Proceedings of the 2017 Innovations in Power and Advanced Computing Technologies (i-PACT), Vellore, India.10.1109/IPACT.2017.8245081 ↗
12. Tolontan, M., Orhei, C., and Vasiu, R. (2017, January 21–22). Overview of the Technical Risk Analysis and Risk Assessment Analysis in Automotive Industry. Proceedings of the 2024 International Symposium on Electronics and Telecommunications (ISETC), Vellore, India.10.1109/ISETC63109.2024.10797338 ↗
13. (2018). Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018). Available online: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en.
14. Hassenzahl, M. (2008, January 2–5). User experience (UX): Towards an experiential perspective on product quality. Proceedings of the 20th Conference on l’Interaction Homme-Machine, in IHM ’08, Metz, France.10.1145/1512714.1512717 ↗
15. Lopata, A., Butkienė, R., Gudonienė, D., and Sukackė, V. (2020). Standardised Questionnaires in Usability Evaluation. Applying Standardised Usability Questionnaires in Digital Products Evaluation. Information and Software Technologies, Springer.
16. Rotaru, O., Vert, S., and Vasiu, R. (2025, January 4–6). Usability Evaluation of Requirement Collaboration Features in Requirements Management Tools. Proceedings of the 27th International Conference on Enterprise Information Systems, Porto, Portugal. Available online: https://www.scitepress.org/Link.aspx?doi=10.5220/0013281800003929.10.5220/0013281800003929 ↗
17. Vert, S., Rotaru, O., Andone, D., Antonica, M., Borobar, A., Orhei, C., and Holotescu, V. (2023). Towards User Experience Guidelines for Mobile Augmented Reality Storytelling with Historical Images in Urban Cultural Heritage. Extended Reality and Metaverse, Springer.10.1007/978-3-031-25390-4_13 ↗
18. Marcus, A., Rosenzweig, E., and Soares, M.M. (2024). Qualitative and Quantitative Approaches to Conduct Heuristic Evaluations: A Comparison Study. Design, User Experience, and Usability, Springer Nature.
19. (2025, August 21). 10 Usability Heuristics for User Interface Design. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/ten-usability-heuristics/.
20. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Avila, R. (2024). GPT-4 Technical Report. arXiv.
21. Open AI (2025, August 24). GPT-5 System Card. Available online: https://cdn.openai.com/gpt-5-system-card.pdf.
22. Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., and Silver, D. (2025). Gemini: A Family of Highly Capable Multimodal Models. arXiv.
23. Sahoo, P., Singh, A.K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2025). A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv.
24. The evolution of embedded systems in automotive industry: A global review10.30574/wjarr.2024.21.2.0420 ↗
25. Mechanical engineering in automotive innovation: A review of electric vehicles and future trends10.30574/ijsra.2024.11.1.0081 ↗
26. Tolontan, M., Orhei, C., and Vasiu, R. (2025, January 22–25). Automotive Specific AI Based Tools for Technical Risk Analysis and Risk Assessment Analysis. Proceedings of the 2025 IEEE 31st International Symposium for Design and Technology in Electronic Packaging (SIITME), Brasov, Romania.10.1109/SIITME67657.2025.11293568 ↗
27. Xu, T., Zhou, Z., Chen, X., Xue, Z., Zhao, Y., Zhang, M., and Jin, Z. (2025, January 1–5). Requirements Dependency Driven Test Case Generation: An Automotive Industry Practice. Proceedings of the 2025 IEEE 33rd International Requirements Engineering Conference (RE), Valencia, Spain.10.1109/RE63999.2025.00046 ↗
28. Zolfaghari, V., Petrovic, N., Pan, F., Lebioda, K., and Knoll, A. (2024, January 26–29). Adopting RAG for LLM-Aided Future Vehicle Design. Proceedings of the 2024 2nd International Conference on Foundation and Large Language Models (FLLM), Dubai, United Arab Emirates.10.1109/FLLM63129.2024.10852467 ↗
29. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv.
30. Chen, J., Li, J., Peng, Z., Wang, W., Ren, Y., Shi, L., and Hu, X. (2025). LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning. arXiv.
31. Flores, F., Mora, M., Alvarez, F., O’Connor, R., and Macias-Luevano, J. (2009). Requirements Engineering: A Review of Processes and Techniques. Handbook of Research on Modern Systems Analysis and Design Technologies and Applications, IGI Global.10.4018/978-1-59904-887-1.ch006 ↗
32. Product lifecycle management for automotive development focusing on supplier integration10.1016/j.compind.2007.07.002 ↗
33. (2025, May 03). UNECE—ECE/TRANS/WP29/2016/2—Agreement Concerning the Adoption of Harmonized Technical United Nations Regulations for Wheeled Vehicles, Equipment and Parts Which Can be Fitted and/or be Used on Wheeled Vehicles and the Conditions for Reciprocal Recognition of Approvals Granted on the Basis of these United Nations Regulations. Available online: https://unece.org/wp29-introduction.
34. Staron, M. (2017). Software Architectures: Views and Documentation. Automotive Software Architectures: An Introduction, Springer International Publishing.10.1007/978-3-319-58610-6 ↗
35. Bellairs, R. (2024, September 30). What Is ISO 26262? Overview + ASIL | Perforce. Available online: https://www.perforce.com/blog/qac/what-is-iso-26262.
36. Yilmaz, M., Clarke, P., Riel, A., Messnarz, R., Greiner, C., and Peisl, T. (2024). Analysis of Automotive SPICE V4.0 Changes. Systems, Software and Services Process Improvement, Springer Nature.10.1007/978-3-031-71142-8 ↗
37. In-Depth Exploration of ISO/SAE 21434 and Its Correlations with Existing Standards10.1109/MCOMSTD.0001.2100080 ↗
38. The state-of-practice in requirements specification: An extended interview study at 12 companies10.1007/s00766-023-00399-7 ↗
39. Cleland-Huang, J., Gotel, O., and Zisman, A. (2012). Traceability Fundamentals. Software and Systems Traceability, Springer.10.1007/978-1-4471-2239-5 ↗
40. Baillargeon, R. (2025, May 21). Managing Changes with Configurations in DOORS Next. Available online: https://www.sodiuswillert.com/en/blog/recommendations-on-using-changes-with-configurations-in-doors-next.
41. Jain, A., and Solutions, V. (2025, September 03). What is Requirements Versioning: Definition, Best Tools & Practices. Available online: https://visuresolutions.com/alm-guide/requirements-versioning/.
42. Ma, J., Wang, G., Lu, J., Vangheluwe, H., Kiritsis, D., and Yan, Y. (2022). Systematic Literature Review of MBSE Tool-Chains. Appl. Sci., 12.10.3390/app12073431 ↗
43. Usability evaluation methods: A literature review
44. Cognitive engineering principles for enhancing human-computer performance10.1080/10447319609526147 ↗
45. Ergonomic criteria for evaluating the ergonomic quality of interactive systems10.1080/014492997119806 ↗
46. Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S., Elmqvist, N., and Diakopoulos, N. (2016). Designing the User Interface: Strategies for Effective Human-Computer Interaction, Pearson. [6th ed.].
47. Moran, K., and Evaluations, K.G.H. (2025, February 06). How to Conduct a Heuristic Evaluation. Available online: https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/.
48. Nielsen, J. (2025, June 02). Severity Ratings for Usability Problems: Article by Jakob Nielsen. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/.
49. A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges10.1109/ACCESS.2024.3365742 ↗
50. A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students’ Formative Assessment Responses in Science
51. Yang, X.-S., Sherratt, S., Dey, N., and Joshi, A. (2024, January 19–22). Switching from DOORS to DOORS Next Generation. Identifying the Main Usability Issues. A Case Study. Proceedings of the Ninth International Congress on Information and Communication Technology, London, UK.
52. SUS: A retrospective
53. Sauro, J. (2025, May 30). Measuring Usability with the System Usability Scale (SUS)—MeasuringU. Available online: https://measuringu.com/sus/.
54. Rotaru, O., Vert, S., and Vasiu, R. (2023, January 27–28). Usability Evaluation of Requirements Collaboration Features in RM Tools. 2023 International Conference on Engineering and Emerging Technologies, Istanbul, Turkey.
55. Moran, K. (2025, February 06). Usability (User) Testing 101. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/usability-testing-101/.
56. Heuristic Usability Evaluation: A Case Study of Online Enrolment System of a State University10.22161/ijaers.86.44 ↗
57. Mahfudz, M.S., Agusti, F., Zahra, S.A., and Dhini, B.R. (2022, January 17). Heuristic evaluation analysis using the 10 Nielsen rule usability method on the KAI Access application. Proceeding of the 3rd International Conference Health, Science And Technology (ICOHETECH), Surakarta, Indonesia.
58. Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A.J., Welihinda, A., Hayes, A., and Radford, A. (2024). GPT-4o System Card. arXiv.
59. Guerino, G., Rodrigues, L., Capeleti, B., Ferreira, R., Freire, A., and Zaina, L. (2025). Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation. arXiv.10.1007/978-3-032-05005-2_20 ↗
60. Wang, S., Hu, M., Li, Q., Safari, M., and Yang, X. (2025). Capabilities of GPT-5 on Multimodal Medical Reasoning. arXiv.
61. Liu, X., and Ni, Z. (2024). Role-playing Prompt Framework: Generation and Evaluation. arXiv.
62. Zhao, Y., Wang, Z., Jung, C., and Zhang, Z. (2024). Large Language Model-Driven Structured Output: A Comprehensive Benchmark and Spatial Data Generation Framework. ISPRS Int. J. Geo-Inf., 13.10.3390/ijgi13110405 ↗
63. Braun, V., and Clarke, V. (2021). Thematic Analysis: A Practical Guide, SAGE Publications Ltd.. Available online: https://www.torrossa.com/it/resources/an/5282292.
64. Ahmed, A., and Imran, A.S. (2025). The role of large language models in UI/UX design: A systematic literature review. arXiv.
65. Chen, Z., Liu, Y., Shi, L., Wang, Z.-J., Chen, X., Zhao, Y., and Ren, F. (May, January 28). MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models. Proceedings of the ACM on Web Conference 2025, Sydney, Australia.10.1145/3696410.3714674 ↗
66. Chiruzzo, L., Ritter, A., and Wang, L. (2025). ScreenQA: Large-Scale Question-Answer Pairs Over Mobile App Screenshots. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Association for Computational Linguistics.