Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection
Source abstract
Abstract
Voice- and gaze-based hands-free input are increasingly used in human-machine interaction. Attempts to combine them into a hybrid technology typically employ the voice channel as an information-rich channel. Voice seems to be “overqualified” to serve simply as a substitute of a computer mouse click, to confirm selections made by gaze. It could be expected that the user would feel discomfort if they had to frequently make “clicks” using their voice, or easily get bored, which also could lead to low performance. To test this, we asked 23 healthy participants to select moving objects with smooth pursuit eye movements. Manual confirmation of selection was faster and rated as more convenient than voice-based confirmation. However, the difference was not high, especially when voice was used to pronounce objects’ numbers (speech recognition was not applied): Score of convenience (M ± SD) was 9.2 ± 1.1 for manual and 8.0 ± 2.1 for voice confirmation, and time spent per object was 1269 ± 265 ms and 1626 ± 331 ms, respectively. We conclude that “voice-as-click” can be used to confirm selection in gaze-based interaction with computers as a substitute for the computer mouse click when manual confirmation cannot be used.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Gaze Tracking and Assistive Technology · Tactile and Sensory Interactions · EEG and Brain-Computer Interfaces
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Laboratory for Neurocognitive Technologies, NRC “Kurchatov Institute”, 123182 Moscow, Russia
Center of Cognitive Programs and Technologies, The Russian State University for the Humanities, 125993 Moscow, Russia
MEG Center, Moscow State University of Psychology and Education, 123290 Moscow, Russia
Funding
Russian Science Foundation · Russian Foundation for Basic Research
References (33)
Show source-provided references
1. Fairclough, S.H., and Gilleade, K. (2014). Eye Tracking and Eye-Based Human–Computer Interaction. Advances in Physiological Computing, Springer.10.1007/978-1-4471-6392-3 ↗
2. Liebling, D.J., and Dumais, S.T. (2014, January 13–17). Gaze and mouse coordination in everyday work. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing—UbiComp ’14 Adjunct, Seattle, DC, USA.10.1145/2638728.2641692 ↗
3. Velichkovsky, B., Sprenger, A., and Unema, P. (1997). Towards gaze-mediated interaction: Collecting solutions of the “Midas touch problem”. Human-Computer Interaction INTERACT ’97, Springer Science and Business Media LLC.10.1007/978-0-387-35175-9_77 ↗
4. Jacob, R.J.K. (1990, January 1–5). What you look at is what you get: Eye movement-based interaction techniques. Proceedings of the Proceedings of the 27th International Conference on Human Factors in Computing Systems—CHI 09, Seattle, WA, USA.10.1145/97243.97246 ↗
5. Electronic Health Record Interactions through Voice: A Review10.1055/s-0038-1666844 ↗
6. Touchless interaction with software in interventional radiology and surgery: A systematic literature review10.1007/s11548-016-1480-6 ↗
7. Evaluation of Tooth-Click Triggering and Speech Recognition in Assistive Technology for Computer Access10.1177/1545968309341647 ↗
8. Alexa, Siri, Cortana, and More: An Introduction to Voice Assistants10.1080/02763869.2018.1404391 ↗
9. Baldwin, D.A. (1995). Understanding the Link Between Joint Attention and Language. Joint Attention: Its Origins and Role in Development, Erlbaum.
10. Multimodal resources for turn-taking10.1177/1461445607075346 ↗
11. I see what you’re saying: Voice signals influence children’s judgments of direct and averted gaze10.1016/j.jecp.2013.07.005 ↗
12. User interaction in hands-free gaming: A comparative study of gaze-voice and touchscreen interface control10.3906/elk-1710-128 ↗
13. Van Der Kamp, J., and Sundstedt, V. (2011, January 25–29). Gaze and voice controlled drawing. Proceedings of the 1st Conference on Integrating Technology into Computer Science Education—ITiCSE ’96, Darmstadt, Germany.10.1145/1983302.1983311 ↗
14. Wilcox, T., Evans, M., Pearce, C., Pollard, N., and Sundstedt, V. (2008, January 11–15). Gaze and voice based game interaction: The revenge of the killer penguins. Proceedings of the SIGGRAPH’08—ACM SIGGRAPH 2008 Posters, Los Angeles, CA, USA.10.1145/1400885.1400972 ↗
15. Speech-augmented eye gaze interaction with small closely spaced targets10.1145/1117309.1117345 ↗
16. Castellina, E., Corno, F., and Pellegrino, P. (2008, January 26–28). Integrated speech and gaze control for realistic desktop environments. Proceedings of the 2008 Symposium on Eye Tracking Research & Applications—ETRA ’08, Savannah, GA, USA.10.1145/1344471.1344492 ↗
17. Beelders, T.R., and Blignaut, P. (2011). The Usability of Speech and Eye Gaze as a Multimodal Interface for a Word Processor. Speech Technologies, IntechOpen.10.5772/16604 ↗
18. Beelders, T.R., and Blignaut, P.J. (2005, January 13–17). Using eye gaze and speech to simulate a pointing device. Proceedings of the Proceedings of the Symposium on Applied Computing—SAC ’17, Santa Fe, NM, USA.
19. Elepfandt, M., and Grund, M. (2012, January 26). Move it there, or not?: The design of voice commands for gaze with speech. Proceedings of the 4th Workshop on Eye Gaze in Intelligent Human Machine Interaction, Gaze-In 2012, Santa Monica, CA, USA.10.1145/2401836.2401848 ↗
20. Sengupta, K., Ke, M., Menges, R., Kumar, C., and Staab, S. (2018, January 14–17). Hands-free web browsing: Enriching the user experience with gaze and voice modality. Proceedings of the Eye Tracking Research and Applications Symposium (ETRA), Warsaw, Poland.10.1145/3204493.3208338 ↗
21. Communicating attention: Gaze position transfer in cooperative problem solving10.1075/pc.3.2.02vel ↗
22. Isachenko, A.V., Zhao, D.G., Melnichuk, E.V., Dubynin, I.A., Velichkovsky, B.M., and Shishkin, S.L. (2018, January 7–10). The Pursuing Gaze Beats Mouse in Non-Pop-Out Target Selection. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.10.1109/SMC.2018.00595 ↗
23. Vidal, M., Bulling, A., and Gellersen, H. (2013, January 8–12). Pursuits: Spontaneous interaction with displays based on smooth pursuit eye movement and moving targets. Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computin, Zurich, Switzerland.10.1145/2493432.2493477 ↗
24. Esteves, A., Velloso, E., Bulling, A., and Gellersen, H. (2015, January 8–11). Orbits: Gaze interaction for smart watches using smooth pursuit eye movements. Proceedings of the UIST 2015—Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, Charlotte, NC, USA.10.1145/2807442.2807499 ↗
25. Lutz, O.H.M., Venjakob, A.C., and Ruff, S. (2015). SMOOVS: Towards calibration-free text entry by gaze using smooth pursuit movements. J. Eye Mov. Res., 8.10.16910/jemr.8.1.2 ↗
26. Cymek, D.H., Ruff, S., Hofmann, S., Venjakob, A.C., Lutz, O.H.M., and Roetting, M. (2014). Entering PIN codes by smooth pursuit eye movementsc. J. Eye Mov. Res., 7.10.16910/jemr.7.4.1 ↗
27. Exploiting Eye Tracking for Smartphone Authentication
28. Khamis, M., Alt, F., and Bulling, A. (2015, January 7–11). A field study on spontaneous gaze-based interaction with a public display using pursuits. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing—UbiComp ’15, Osaka, Japan.10.1145/2800835.2804335 ↗
29. Lander, C. (2016). Methods for calibration free and multi-user eye tracking. Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, Association for Computing Machinery (ACM).10.1145/2957265.2963116 ↗
30. Abdrabou, Y., Mostafa, M., Khamis, M., and Elmougy, A. (2019, January 25–28). Calibration-free text entry using smooth pursuit eye movements. Proceedings of the 11th ACM Symposium on Eye Tracking Research & Application, Denver, CO, USA.10.1145/3314111.3319838 ↗
31. Voelker, S., Hueber, S., Holz, C., Remy, C., and Marquardt, N. (2020, January 25–30). GazeConduits: Calibration-Free Cross-Device Collaboration through Gaze and Touch. Proceedings of the Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3313831.3376578 ↗
32. hEYEbrid: A hybrid approach for mobile calibration-free gaze estimation10.1145/3161166 ↗
33. Calibration-free gaze interfaces based on linear smooth pursuit10.16910/jemr.13.1.3 ↗