HEUXIVA: A Set of Heuristics for Evaluating User eXperience with Voice Assistants
Source abstract
Abstract
Voice assistants have become increasingly common in everyday devices such as smartphones and smart speakers. Improving their user experience (UX) is crucial to ensuring usability, acceptance, and long-term effectiveness. Heuristic evaluation is a widely used method for UX evaluation due to its efficiency in detecting problems quickly and at low cost. Nonetheless, existing usability/UX heuristics were not designed to address the specific challenges of voice-based interaction, which relies on spoken dialog and auditory feedback. To overcome this limitation, we developed HEUXIVA, a set of 13 heuristics specifically developed for evaluating UX with voice assistants. The proposal was created through a structured methodology and refined in two iterations. We validated HEUXIVA through heuristic evaluations, expert judgment, and user testing. The results offer preliminary but consistent evidence supporting the effectiveness of HEUXIVA in identifying UX issues specific to the voice assistant “Google Nest Mini”. Experts described the heuristics as clear, practical, and easy to use. They also highlighted their usefulness in evaluating interaction features and supporting the overall UX evaluation process. HEUXIVA therefore provides designers, researchers, and practitioners with a specialized tool to improve the quality of voice assistant interfaces and improve user satisfaction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: AI in Service Interactions · Speech and dialogue systems · Social Robot Interaction and HRI
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Valparaíso 2340000, Chile
Departamento de Electrotecnia e Informática, Universidad Técnica Federico Santa María, Viña del Mar 2520000, Chile
Funding
Agencia Nacional de Investigación y Desarrollo
References (41)
Show source-provided references
1. Nielsen, J., and Molich, R. (1990, January 1–5). Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Empowering People—CHI ’90, Seattle, WA, USA.10.1145/97243.97281 ↗
2. A methodology to develop usability/user experience heuristics10.1016/j.csi.2018.03.002 ↗
3. Voice assistant vs. Chatbot–examining the fit between conversational agents’ interaction modalities and information search tasks10.1007/s10796-021-10226-5 ↗
4. Intelligent personal assistants based on internet of things approaches10.1109/JSYST.2016.2555292 ↗
5. An IoT-based mobile gateway for intelligent personal assistants on mobile health environments10.1016/j.jnca.2016.03.014 ↗
6. Understanding adoption of intelligent personal assistants: A parasocial relationship perspective10.1108/IMDS-05-2017-0214 ↗
7. Investigating the use of speech-based conversational agents for life coaching10.1016/j.ijhcs.2021.102745 ↗
8. PAVAL: A location-aware virtual personal assistant for retrieving geolocated points of interest and location-based services10.1016/j.engappai.2018.09.013 ↗
9. Cowan, B.R., Pantidi, N., Coyle, D., Morrissey, K., Clarke, P., Al-Shehri, S., Earley, D., and Bandeira, N. (2017, January 4–7). “What can i help you with?” infrequent users’ experiences of intelligent personal assistants. Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, Vienna, Austria.10.1145/3098279.3098539 ↗
10. Langevin, R., Lordon, R.J., Avrahami, T., Cowan, B.R., Hirsch, T., and Hsieh, G. (2021, January 8–13). Heuristic evaluation of conversational agents. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Virtual.10.1145/3411764.3445312 ↗
11. Towards a set of heuristics for evaluating chatbots10.1109/TLA.2021.9480145 ↗
12. Zwakman, D.S., Pal, D., Triyason, T., and Vanijja, V. (2020, January 21–23). Usability of voice-based intelligent personal assistants. Proceedings of the 2020 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Republic of Korea.10.1109/ICTC49870.2020.9289550 ↗
13. (2025, January 15). Google Actions on Google Glossary (Dialogflow). Available online: https://developers.google.com/assistant/df-asdk/glossary.
14. (2025, January 15). Google Nest Explore What You Can Do with Google Nest and Home Devices. Available online: https://support.google.com/googlenest/answer/7130274.
15. (2025, January 15). Google Nest Customize Smart Plug or Smart Switch Voice Commands with Device Type. Available online: https://support.google.com/googlenest/answer/9921419.
16. (2025, January 15). Google Nest Customize Your News Experience. Available online: https://support.google.com/googlenest/answer/7551674.
17. (2025, January 15). Google Nest Guests and Your Google Connected Home Devices. Available online: https://support.google.com/googlenest/answer/7177221.
18. Development of new commands for Google Assistant using Dialogflow, Firebase and NodeMCU (ESP8266) as an intermediary
19. (2025, January 15). Google Nest FAQs on Privacy: Google Nest. Available online: https://support.google.com/googlenest/answer/9415830.
20. (2025, January 15). Google Assistant What It Can Do—Get Started. Available online: https://assistant.google.com/learn/.
21. (2025, January 15). Google Assistant Control Smart Home Devices with Google Assistant. Available online: https://support.google.com/assistant/answer/7314909?.
22. (2019). Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems (Standard No. ISO 9241-210:2019).
23. Developing elements of user experience for mobile phones and services: Survey, interview, and observation approaches10.1002/hfm.20316 ↗
24. Lykke, M., and Jantzen, C. (2016, January 13–17). User experience dimensions: A systematic approach to experiential qualities for evaluating information interaction in museums. Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, Chapel Hill, NC, USA.10.1145/2854946.2854965 ↗
25. Morville, P. (2025, January 15). User Experience Design. Semantic Studios. Available online: https://semanticstudios.com/user_experience_design/.
26. Usability: Lessons Learned. and Yet to Be Learned10.1080/10447318.2014.930311 ↗
27. Kendrick, A. (2025, January 15). Formative vs. Summative Evaluations. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/formative-vs-summative-evaluations/.
28. Nielsen, J. (2025, January 15). Thinking Aloud: The #1 Usability Tool. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/.
29. (2025, January 15). Experience Research Society UX Expert Evaluation. Available online: https://experienceresearchsociety.org/ux-methods/ux-expert-evaluation/.
30. Harley, A. (2025, January 15). UX Expert Reviews. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/ux-expert-reviews/.
31. Nielsen, J. (2025, January 15). 10 Usability Heuristics for User Interface Design. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/ten-usability-heuristics/.
32. SUS-A quick and dirty usability scale
33. Zwakman, D.S., Pal, D., Triyason, T., and Arpnikanondt, C. (2020, January 14–16). Voice usability scale: Measuring the user experience with voice assistants. Proceedings of the 2020 IEEE International Symposium on Smart Electronic Systems (iSES) (Formerly iNiS), Chennai, India.10.1109/iSES50453.2020.00074 ↗
34. Nielsen, J. (2025, January 15). Severity Ratings for Usability Problems. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/.
35. (2018). Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018). Available online: https://www.iso.org/standard/63500.html.
36. Nielsen, J. (2025, January 15). Usability 101: Introduction to Usability. Nielsen Norman Group. Available online: https://www.nngroup.com/articles/usability-101-introduction-to-usability/.
37. Nowacki, C., Gordeeva, A., and Lizé, A.-H. (2020, January 19–24). Improving the usability of voice user interfaces: A new set of ergonomic criteria. Proceedings of the Design, User Experience, and Usability. Design for Contemporary Interactive Environments: 9th International Conference, DUXU 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark. Proceedings, Part I.10.1007/978-3-030-49760-6_8 ↗
38. (2025, January 15). Google Store Nest Mini—Overview. Available online: https://store.google.com/us/product/google_nest_mini?hl=en-US.
39. Ergonomic criteria for evaluating the ergonomic quality of interactive systems10.1080/014492997119806 ↗
40. Intelligent personal assistants: A systematic literature review10.1016/j.eswa.2020.113193 ↗
41. UXH-GEDAPP: A set of user experience heuristics for evaluating generative design applications10.1016/j.infsof.2024.107408 ↗