Evaluating UX and Usability in Automotive Human–Machine Interfaces: A Systematic Review
Source abstract
Abstract
Human–Machine Interfaces (HMIs) are increasingly important in vehicles and other safety-critical systems, yet approaches to their usability and User eXperience (UX) evaluation remain fragmented. This systematic literature review investigates how HMIs are empirically evaluated across domains, with a primary focus on automotive HMIs, complemented by evidence from related safety-critical domains. The review examines UX and usability evaluation methodologies, tools, standards, and technological trends reported in recent research. Peer-reviewed journal articles published between 2015 and 2025 were considered if they addressed empirical usability or UX evaluation of HMIs. Searches were conducted in Scopus and ScienceDirect databases following PRISMA guidelines. From n = 659 records initially identified, n = 82 papers were included in the final analysis. The literature was synthesized using a descriptive and narrative approach, focusing on evaluation contexts, testing methodologies, sensor-based tools, applied standards, and assessment metrics. Most papers investigated automotive HMIs, while fewer addressed aerospace, industrial, maritime, and other safety-critical applications. Simulation-based user testing emerged as the dominant evaluation approach, frequently supported by eye-tracking and physiological sensing technologies and subjective evaluation questionnaires. A more detailed analysis revealed that adherence to international standards (e.g., ISO 9241 and ISO 26262) was not always consistently evident. Overall, the evidence highlights substantial methodological heterogeneity, fragmented adoption of standards, and limited cross-domain comparability. While today UX and usability evaluation can benefit from continuous technological advances, the field lacks standardized and replicable assessment protocols. Future research should prioritize stronger integration of standards, multimodal evaluation approaches, and longitudinal study designs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human-Automation Interaction and Safety · Ergonomics and Musculoskeletal Disorders · Gaze Tracking and Assistive Technology
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Industrial Engineering, University of Bologna, Viale del Risorgimento 2, 40136 Bologna, Italy
Department of Industrial Engineering, University of Bologna, Via Montaspro 97, 47121 Forlì, Italy
References (56)
Show source-provided references
1. Developments in the Human Machine Interface Technologies and Their Applications: A Review10.1080/03091902.2021.1936237 ↗
2. Grobelna, I., Mailland, D., and Horwat, M. (2025). Design of Automotive HMI: New Challenges in Enhancing User Experience, Safety, and Security. Appl. Sci., 15.10.3390/app15105572 ↗
3. Distraction Pattern Classification and Comparisons under Different Conditions in the Full-Touch HMI Mode10.1016/j.displa.2023.102413 ↗
4. Nielsen, J., and Molich, R. (1990, January 1–5). Heuristic Evaluation of User Interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA, USA.10.1145/97243.97281 ↗
5. (2025, May 20). ISO 9241-210:2010—Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems. Available online: https://www.iso.org/standard/52075.html.
6. (2025, May 20). ISO 26262-1:2018—Road Vehicles—Functional Safety—Part 1: Vocabulary. Available online: https://www.iso.org/standard/68383.html.
7. (2018). Ergonomics of Human-System Interaction-Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018).
8. Robert, J., and Lesage, A. (2017). Designing and Evaluating User Experience. The Handbook of Human-Machine Interaction, CRC Press.10.1201/9781315557380-16 ↗
9. Sustainable and Flexible Industrial Human Machine Interfaces to Support Adaptable Applications in the Industry 4.0 Paradigm10.1080/00207543.2019.1572932 ↗
10. Zhang, X., Liao, X.-P., Tu, J.-C.A., Zhang, X., Liao, X.-P., and Tu, J.-C. (2022). A Study of Bibliometric Trends in Automotive Human–Machine Interfaces. Sustainability, 14.10.3390/su14159262 ↗
11. Multi-Modal Interaction between Pilots and Avionic Systems on-Board Large Commercial Aircraft
12. Gauci, J., Cauchi, N., Theuma, K., Zammit-Mangion, D., and Muscat, A. (2015, January 13–17). Design and Evaluation of a Touch Screen Concept for Pilot Interaction with Avionic Systems. Proceedings of the 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC), Prague, Czech Republic.10.1109/DASC.2015.7311566 ↗
13. Roto, V., Väätäjä, H., Jumisko-Pyykkö, S., and Väänänen-Vainio-Mattila, K. (2011, January 28–30). Best Practices for Capturing Context in User Experience Studies in the Wild. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, Tampere, Finland.10.1145/2181037.2181054 ↗
14. Hornbæk, K., and Law, E.L.C. (May, January 28). Meta-Analysis of Correlations among Usability Measures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.
15. Usability and User Experience Evaluation in Intelligent Environments: A Review and Reappraisal
16. Nielsen, J. (1994, January 24–28). Enhancing the Explanatory Power of Usability Heuristics. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.10.1145/191666.191729 ↗
17. Dumas, J.S., and Redish, J.C. (1999). A Practical Guide to Usability Testing, Intellect Books.
18. Hart, C. (1998). Doing a Literature Review: Releasing the Research Imagination, Sage.
19. The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews10.1136/bmj.n71 ↗
20. Ecosystem Services Research in Mountainous Regions: A Systematic Literature Review on Current Knowledge and Research Gaps10.1016/j.scitotenv.2019.134581 ↗
21. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement10.1186/2046-4053-4-1 ↗
22. Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review10.1111/1467-8551.00375 ↗
23. Ahn, D.K., Bae, B.C., and Kim, Y. (2023). User Experience of a Digital Fashion Show: Exploring the Effectiveness of Interactivity in Virtual Reality. Appl. Sci., 13.10.3390/app13042558 ↗
24. Improving the Experience in the Pedestrian’s Interaction with an Autonomous Vehicle: An Ergonomic Comparison of External HMI10.1016/j.apergo.2021.103478 ↗
25. Usability Testing of Three Visual HMIs for Assisted Driving: How Design Impacts Driver Distraction and Mental Models10.1080/00140139.2022.2136766 ↗
26. Smart and Cognitive Solutions for Operator 4.0: Laboratory H-CPPS Case Studies10.1016/j.cie.2018.10.032 ↗
27. Creating Trust in Fully Autonomous Vehicles through Spatialized Predictive Sonification10.1016/j.ijhcs.2025.103481 ↗
28. Do Users Desire Gestures for In-Vehicle Interaction? Towards the Subjective Assessment of Gestures in a High-Fidelity Driving Simulator10.1016/j.chb.2024.108189 ↗
29. Performance and Visual Appearance of In-Vehicle Voice Assistants Impact User Experience: A Comparative Study between Chinese and German Users10.1016/j.trf.2025.01.041 ↗
30. A Human Factors Perspective on How to Keep SAE Level 3 Conditional Automated Driving Safe
31. An Immersive Spatially Consistent Multi-Modal Augmented Virtuality Human-Machine Interface for Telerobotic Systems10.1016/j.compind.2025.104260 ↗
32. Usability Testing of Novel IoT-Infused Digital Services on Farm Equipment Reveals Farmer’s Requirements towards Future Human–Machine Interface Design Guidelines10.3390/agriengineering6020095 ↗
33. Evaluating Secondary Input Devices to Support an Automotive Touchscreen HMI: A Cross-Cultural Simulator Study Conducted in the UK and China10.1016/j.apergo.2019.03.005 ↗
34. Design of Ergonomic Dashboards for Tractors and Trucks: Innovative Method and Tools
35. Human Factors in Developing Automated Vehicles: A Requirements Engineering Perspective10.1016/j.jss.2023.111810 ↗
36. Haptic Experience Design: What Hapticians Do and Where They Need Help10.1016/j.ijhcs.2017.04.004 ↗
37. Human-Centred Data-Driven Redesign of Simulation-Based Training: A Qualitative Study Applied on Two Use Cases of the Healthcare and Industrial Domains
38. Effects of Input Method and Display Mode of Situation Map on Early Warning Aircraft Reconnaissance Task Performance with Different Information Complexities10.1016/j.cja.2022.06.011 ↗
39. Designing Interactive Glazing through an Engineering Psychology Approach: Six Augmented Reality Scenarios That Envision Future Car Human-Machine Interface10.1016/j.vrih.2022.07.004 ↗
40. Improving Road User Perception in Adverse Weather: An Augmented Human–Machine Interface for Remote Assistants of Automated Vehicles10.1016/j.trf.2025.05.017 ↗
41. Gorin, H., Patel, J., Qiu, Q., Merians, A., Adamovich, S., and Fluet, G. (2024). A Review of the Use of Gaze and Pupil Metrics to Assess Mental Workload in Gamified and Simulated Sensorimotor Tasks. Sensors, 24.10.3390/s24061759 ↗
42. An Overview of Approaches and Methods for the Cognitive Workload Estimation in Human–Machine Interaction Scenarios through Wearables Sensors10.3390/biomedinformatics4020064 ↗
43. Masi, G., Amprimo, G., Rechichi, I., Ferraris, C., and Olmo, G. (2024, January 15–19). Does Baseline Stress Affect Electrodermal Activity? A Serious-Game-Based Pilot Study. Proceedings of the 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.10.1109/EMBC53108.2024.10781944 ↗
44. SUS-a Quick and Dirty Usability Scale
45. An Empirical Evaluation of the System Usability Scale10.1080/10447310802205776 ↗
46. Usability Assessment of Compaction Operator Support Systems Using Virtual Prototyping10.1016/j.autcon.2021.103784 ↗
47. Sogemeier, D., Forster, Y., Naujoks, F., Krems, J.F., and Keinath, A. (2024). Driving across Markets: An Analysis of a Human–Machine Interface in Different International Contexts. Information, 15.10.3390/info15060349 ↗
48. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research10.1016/S0166-4115(08)62386-9 ↗
49. Human-Centered Design and Evaluation of a Workplace for the Remote Assistance of Highly Automated Vehicles10.1007/s10111-024-00753-x ↗
50. Applying the User Experience Questionnaire (UEQ) in Different Evaluation Scenarios
51. Burkard, A., Zimmermann, G., and Schwarzer, B. (2021). Monitoring Systems for Checking Websites on Accessibility. Front. Comput. Sci., 3.10.3389/fcomp.2021.628770 ↗
52. BEYOND—Building an Experience to Yield Optimal Nautical Displays—Measuring Nautical Situation Awareness10.1088/1742-6596/2867/1/012044 ↗
53. You, F., Deng, H., Hansen, P., and Zhang, J. (2022). Research on Transparency Design Based on Shared Situation Awareness in Semi-Automatic Driving. Appl. Sci., 12.10.3390/app12147177 ↗
54. The Spatial Presence Experience Scale (SPES): A Short Self-Report Measure for Diverse Media Settings10.1027/1864-1105/a000137 ↗
55. Development of the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST)10.1080/10400435.1996.10132268 ↗
56. Comparative Analysis of SAGAT and SART for Evaluations of Situation Awareness