Eye-Tracking in Interactive Virtual Environments: Implementation and Evaluation
Source abstract
Abstract
Not all eye-tracking methodology and data processing are equal. While the use of eye-tracking is intricate because of its grounding in visual physiology, traditional 2D eye-tracking methods are supported by software, tools, and reference studies. This is not so true for eye-tracking methods applied in virtual reality (imaginary 3D environments). Previous research regarded the domain of eye-tracking in 3D virtual reality as an untamed realm with unaddressed issues. The present paper explores these issues, discusses possible solutions at a theoretical level, and offers example implementations. The paper also proposes a workflow and software architecture that encompasses an entire experimental scenario, including virtual scene preparation and operationalization of visual stimuli, experimental data collection and considerations for ambiguous visual stimuli, post-hoc data correction, data aggregation, and visualization. The paper is accompanied by examples of eye-tracking data collection and evaluation based on ongoing research of indoor evacuation behavior.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Gaze Tracking and Assistive Technology · Virtual Reality Applications and Impacts · Teleoperation and Haptic Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Geography, Faculty of Science, Masaryk University, Kotlářská 267/2, 611 37 Brno, Czech Republic
Department of Educational Sciences, Faculty of Arts, Masaryk University, Arna Nováka 1, 602 00 Brno, Czech Republic
Department of Information and Library Studies, Faculty of Arts, Masaryk University, Arna Nováka 1, 602 00 Brno, Czech Republic
Institute of Psychology, Freie Universität Berlin, Kaiserswerther Straße 16-18, 14195 Berlin, Germany
Funding
Technology Agency of the Czech Republic · Masaryk University
References (100+)
Show source-provided references
1. Gliner, J.A., Morgan, G.A., and Leech, N.L. (2017). Research Methods in Applied Settings: An Integrated Approach to Design and Analysis, Routledge.10.4324/9781315723082 ↗
2. Cacioppo, J.T., and Tassinary, L.G. (1990). Psychometric principles: From physiological data to psychological constructs. Principles of Psychophysiology: Physical, Social, and Inferential Elements, Cambridge University Press.
3. The 35th Sir Frederick Bartlett Lecture: Eye Movements and Attention in Reading, Scene Perception, and Visual Search10.1080/17470210902816461 ↗
4. Intentional response distortion on personality tests: Using eye-tracking to understand response processes when faking10.1037/a0025711 ↗
5. Tsianos, N., Germanakos, P., Lekkas, Z., Mourlas, C., and Samaras, G. (2009, January 15–17). Eye-tracking users’ behavior in relation to cognitive style with inane-learning environment. Proceedings of the Ninth IEEE International Conference on Advanced Learning Technologies, Riga, Latvia.10.1109/ICALT.2009.110 ↗
6. Eye tracking young children with autism
7. Eye movement desensitisation and reprocessing: An update10.1192/apt.11.5.347 ↗
8. Ekman, P. (2015). Emotion in the Human Face, Malor Books.
9. Changing Minds (2021, July 31). Eyes: Body Language. Available online: http://changingminds.org/techniques/body/parts_body_language/eyes_body_language.htm.
10. Yarbus, A.L. (1967). Eye Movements and Vision, Plenum Press.10.1007/978-1-4899-5379-7 ↗
11. Gaze performance during face-to-face communication: A live eye tracking study of typical children and children with autism10.1016/j.rasd.2015.06.007 ↗
12. Eye-tracking and artificial intelligence to enhance motivation and learning10.1186/s40561-020-00122-x ↗
13. Shipman, K. (2021, July 31). MetaHumans—A look past the Hype. Available online: https://www.pugetsystems.com/labs/articles/MetaHumans---a-look-past-the-Hype-2139.
14. Buswell, G.T. (1935). How People Look at Pictures: A Study of the Psychology and Perception in Art, University of Chicago Press.
15. Pioneers of eye movement research10.1068/i0389 ↗
16. Accurate Two-Dimensional Eye Tracker Using First and Fourth Purkinje Images10.1364/JOSA.63.000921 ↗
17. New instrument for monitoring eye fixation and pupil size during the visual field examination10.1007/BF02441846 ↗
18. Lithography for now and the future10.1016/j.sse.2019.03.006 ↗
19. Safin, R., Garipova, E., Lavrenov, R., Li, H., Svinin, M., and Magid, E. (2020, January 23–26). Hardware and software video encoding comparison. Proceedings of the 59th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), Chiang Mai, Thailand.10.23919/SICE48898.2020.9240439 ↗
20. Head-Mounted Eye Gaze Tracking Devices: An Overview of Modern Devices and Recent Advances
21. Ewalt, D.M. (2018). Defying Reality: The inside Story of the Virtual Reality Revolution, Blue Rider Press.
22. Xinxign, T. (2012). The virtual reality revolution: The vision and the reality. Virtual Reality—Human Computer Interaction, InTech.10.5772/3333 ↗
23. Augmented Reality and Virtual Reality Displays: Perspectives and Challenges10.1016/j.isci.2020.101397 ↗
24. To Go Where No Man Has Gone Before: Virtual Reality in Architecture, Landscape Architecture and Environmental Planning10.1016/j.compenvurbsys.2015.05.001 ↗
25. Griliopoulos, D. (2021, September 21). The Virtual Reality of Virtual Worlds—Improbable. Available online: https://www.improbable.io/blog/the-virtual-reality-of-virtual-worlds.
26. A Comparative Study of Virtual Reality Methods of Interaction and Locomotion Based on Presence, Cybersickness and Usability10.1109/TETC.2019.2915287 ↗
27. Why and How to Use Virtual Reality to Study Human Social Interaction: The Challenges of Exploring a New Research Landscape10.1111/bjop.12290 ↗
28. Sussman, A., and Hollander, J.B. (2015). Cognitive Architecture: Designing for How We Respond to the Built Environment, Routledge.10.4324/9781315856964 ↗
29. Eye Tracking for Spatial Research: Cognition, Computation, Challenges10.1080/13875868.2016.1254634 ↗
30. Patney, A., Kim, J., Salvi, M., Kaplanyan, A., Wyman, C., Benty, N., Lefohn, A., and Luebke, D. (2016, January 24–28). Perceptually-Based Foveated Virtual Reality. Proceedings of the ACM SIGGRAPH 2016 Emerging Technologies, Anaheim, CA, USA.10.1145/2929464.2929472 ↗
31. Tobii (2021, August 25). Tobii XR SDK, 2021a. Available online: https://vr.tobii.com/sdk/.
32. Use of Photogrammetry in Video Games: A Historical Overview10.1177/1555412018786415 ↗
33. Yan, J., Kensek, K., Konis, K., and Noble, D. (2020). CFD Visualization in a Virtual Reality Environment Using Building Information Modeling Tools. Buildings, 10.10.3390/buildings10120229 ↗
34. Harnessing the Power of Immersive Virtual Reality—Visualization and Analysis of 3D Earth Science Data Sets10.1080/10095020.2019.1621544 ↗
35. Krejtz, K., Duchowski, A.T., Niedzielska, A., Biele, C., and Krejtz, I. (2018). Eye Tracking Cognitive Load Using Pupil Diameter and Microsaccades with Fixed Gaze. PLoS ONE, 13.10.1371/journal.pone.0203629 ↗
36. Duchowski, A. (2017). Eye Tracking Methodology: Theory and Practice, Springer International Publishing. [3rd ed.].10.1007/978-3-319-57883-5 ↗
37. Eye Tracking in the Wild: The Good, the Bad and the Ugly10.16910/jemr.8.5.1 ↗
38. Clay, V., König, P., and König, S. (2019). Eye Tracking in Virtual Reality. J. Eye Mov. Res., 12.10.16910/jemr.12.1.3 ↗
39. SR Research (2021, October 18). EyeLink 1000 Plus Technical Specifications. Available online: https://www.sr-research.com/wp-content/uploads/2017/11/eyelink-1000-plus-specifications.pdf.
40. Sipatchin, A., Wahl, S., and Rifai, K. (2021). Eye-Tracking for Clinical Ophthalmology with Virtual Reality (VR): A Case Study of the HTC Vive pro Eye’s Usability. Healthcare, 9.10.3390/healthcare9020180 ↗
41. VRgineers (2021, October 25). XTAL Technical Specification. Available online: https://vrgineers.com/xtal/technical-specification/.
42. Varjo (2021, October 25). Getting Started with Varjo Plugin for Unity. Available online: https://developer.varjo.com/docs/v2.1.0/unity/getting-started-with-varjo-plugin-for-unity.10.1007/978-1-4842-7167-4_1 ↗
43. Khronos Group (2021, August 25). The Open XR Specification. Available online: https://www.khronos.org/registry/OpenXR/specs/1.0/pdf/xrspec.pdf.
44. Contributions of Pictorial and Binocular Cues to the Perception of Distance in Virtual Reality10.1007/s10055-021-00500-x ↗
45. Effect of Degrees of Freedom on the Sense of Presence Generated by Virtual Reality (VR) Head-Mounted Display Systems: A Case Study on the Use of VR in Early Design Studios10.1177/0047239518824862 ↗
46. Kitson, A., Hashemian, A.M., Stepanova, E.R., Kruijff, E., and Riecke, B.E. (2017, January 18–19). Comparing leaning-based motion cueing interfaces for virtual reality locomotion. Proceedings of the 2017 IEEE Symposium on 3D User Interfaces, Los Angeles, CA, USA.10.1109/3DUI.2017.7893320 ↗
47. Keil, J., Edler, D., O’Meara, D., Korte, A., and Dickmann, F. (2021). Effects of Virtual Reality Locomotion Techniques on Distance Estimations. ISPRS Int. J. Geo-Inf., 10.10.3390/ijgi10030150 ↗
48. Exploring and Transforming Spaces through High-Dimensional Gestural Interactions10.5194/ica-adv-3-2-2021 ↗
49. Adobe Inc. Medium (2021, December 23). Top 3D Sculpting Tools for Virtual Reality Authoring. Available online: https://www.adobe.com/products/medium.html.
50. Lee, N. (2018). Raycasting in Virtual Reality. Encyclopedia of Computer Graphics and Games, Springer.
51. Meixner, G., and Müller, C. (2017). Virtual Reality Driving Simulator Based on Head-Mounted Displays. Automotive User Interfaces, Springer.10.1007/978-3-319-49448-7 ↗
52. Gillies, M. (2016, January 5–6). What Is Movement Interaction in Virtual Reality For?. Proceedings of the 3rd International Symposium on Movement and Computing, Thessaloniki, Greece.10.1145/2948910.2948951 ↗
53. Unity Technologies (2021, August 25). Character Controller. Available online: https://docs.unity3d.com/Manual/class-CharacterController.html.
54. Signage Visibility Analysis and Optimization System Using BIM-Enabled Virtual Reality (VR) Environments10.1016/j.aei.2017.03.005 ↗
55. Ugwitz, P., Stachoň, Z., and Pospíšil, P. (2020, January 20–25). Rendering a series of 3D dynamic visualizations in (geographic) experimental tasks. Proceedings of the 8th International Conference on Cartography and GIS, Nessebar, Bulgaria.
56. Framework of Affordances for Virtual Reality and Augmented Reality10.1080/07421222.2019.1628877 ↗
57. Pettersson, J., Albo, A., Eriksson, J., Larsson, P., Falkman, K.W., and Falkman, P. (2018, January 12–13). Cognitive Ability Evaluation Using Virtual Reality and Eye Tracking. Proceedings of the 2018 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), Ottawa, ON, Canada.10.1109/CIVEMSA.2018.8439999 ↗
58. Tobii (2021, October 18). Eye-Tracking Analytics. Available online: https://vr.tobii.com/sdk/learn/analytics/.
59. Halladay, K. (2019). Practical Shader Development: Vertex and Fragment Shaders for Game Developers, Apress.10.1007/978-1-4842-4457-9 ↗
60. WorldViz Inc (2021, August 26). Sightlab VR. Available online: https://www.worldviz.com/virtual-reality-eye-tracking-for-research-solutions.
61. Rönngren, D. (2021, July 31). OpenXR Announces the First Extension for Eye Tracking. Available online: https://blog.tobii.com/openxr-announces-the-first-extension-for-eye-tracking.
62. Tobii (2021, October 18). Get Started Developing with the HTC VIVE Pro Eye Using Tobii XR SDK. Available online: https://vr.tobii.com/sdk/develop/unity/getting-started/vive-pro-eye/.
63. Epic Games (2021, October 18). The Most Powerful Real-Time 3D Creation Tool—Unreal Engine. Available online: https://www.unrealengine.com/.
64. Unity Technologies (2021, October 18). Unity Real-Time Development Platform. Available online: https://unity.com/releases/release-overview.
65. Bergsman Lind, T. (2021, September 26). Quixel Joins Forces with Epic Games. Available online: https://quixel.com/blog/2019/11/12/quixel-joins-forces-with-epic-games.
66. Beardsall, R. (2021, September 26). Unreal Engine 5 And Nanite Virtualized Geometry—What Does It Mean For Content Creators?. Available online: https://medium.com/xrlo-extended-reality-lowdown/unreal-engine-5-and-nanite-virtualized-geometry-what-does-it-mean-for-content-creators-b4106accd306.
67. Romero, M.F., and Sewell, B. (2019). Blueprints Visual Scripting for Unreal Engine: The Faster Way to Build Games Using UE4 Blueprints, Packt Publishing.
68. 3D Visualization of Historical Buildings: Methods, Their Utility in the Tourism Industry and Beyond
69. Unity Technologies (2021, August 25). Post-Processing. Available online: https://docs.unity3d.com/Manual/PostProcessingOverview.html.
70. Building Information Modeling (BIM) for existing buildings—Literature review and future needs10.1016/j.autcon.2013.10.023 ↗
71. Understanding and facilitating BIM adoption in the AEC industry10.1016/j.autcon.2010.09.002 ↗
72. Tridify Ltd (2021, August 25). Tridify Convert. Available online: https://www.tridify.com/.
73. Tridify Ltd (2021, August 25). Tridify BIM Tools. Available online: https://assetstore.unity.com/packages/tools/utilities/tridify-bim-tools-125779.
74. Unity Technologies (2021, August 25). ProBuilder. Available online: https://unity.com/features/probuilder.
75. Blender Foundation (2021, August 25). Blender—A 3D Modelling and Rendering Package. Available online: https://www.blender.org/.
76. Luebke, D.P. (2003). Level of Detail for 3D Graphics, Morgan Kaufmann.10.1016/B978-155860838-2/50003-0 ↗
77. Aversa, D., and Dickinson, C. (2019). Unity Game Optimization: Enhance and Extend the Performance of All Aspects of Your Unity Games, Packt Publishing. [3rd ed.].
78. Austin, J. (2021, September 29). Fix your (Unity) Timestep. Available online: https://johnaustin.io/articles/2019/fix-your-unity-timestep.
79. Farrell, J.E. (1991). Fitting Physical Screen Parameters to the Human Eye. The Man-Machine Interface, Macmillan.
80. Lengyel, E. (2016). Foundations of Game Engine Development, Terathon Software LLC.10.1201/b21177 ↗
81. Foveated 3D Graphics10.1145/2366145.2366183 ↗
82. Nystrom, R. (2022, January 16). Game Programming Patterns. Available online: https://gameprogrammingpatterns.com/.
83. Space-Time Visual Analytics of Eye-Tracking Data for Dynamic Stimuli10.1109/TVCG.2013.194 ↗
84. Unreal Engine (2022, January 05). Decals. An Overview of the Decals Example Level. Available online: https://docs.unrealengine.com/4.27/en-US/Resources/ContentExamples/Decals/.
85. Llanes-Jurado, J., Marín-Morales, J., Guixeres, J., and Alcañiz, M. (2020). Development and Calibration of an Eye-Tracking Fixation Identification Algorithm for Immersive Virtual Reality. Sensors, 20.10.3390/s20174956 ↗
86. On Visual Complexity of 3D Shapes10.1016/j.cag.2011.03.006 ↗
87. How Are Three-Dimensional Objects Represented in the Brain?10.1093/cercor/5.3.247 ↗
88. Ikonen, T. (2022, January 05). Lightweight CSV Reader for Unity. Available online: https://bravenewmethod.com/2014/09/13/lightweight-csv-reader-for-unity/.
89. SensoMotoric Instruments (2014). Event Detection. BeGaze Manual Version 3.4, SensoMotoric Instruments.
90. Who Invented Ray Tracing?10.1007/BF01911003 ↗
91. Yamu, C., van Nes, A., and Garau, C. (2021). Bill Hillier’s Legacy: Space Syntax—A Synopsis of Basic Concepts, Measures, and Empirical Application. Sustainability, 13.10.3390/su13063394 ↗
92. Lachambre, S., Lagarde, S., and Jover, C. (2017). Unity Photogrammetry Workflow, Unity Technologies.
93. Fuhl, W., Kuebler, T., Santini, T., and Kasneci, E. (2018, January 10–12). Automatic Generation of Saliency-Based Areas of Interest for the Visualization and Analysis of Eye-Tracking Data. Proceedings of the Conference on Vision, Modeling, and Visualization, Stuttgart, Germany.
94. A Benchmark of Four Methods for Generating 360 Saliency Maps from Eye Tracking Data10.1142/S1793351X19400142 ↗
95. Soler-Dominguez, J.L., Camba, J.D., Contero, M., and Alcañiz, M. (2017). A Proposal for the Selection of Eye-Tracking Metrics for the Implementation of Adaptive Gameplay in Virtual Reality Based Games. International Conference on Virtual, Augmented and Mixed Reality, Springer.10.1007/978-3-319-57987-0_30 ↗
96. Toggle Toolkit: A Tool for Conducting Experiments in Unity Virtual Environments10.3758/s13428-020-01510-4 ↗
97. Ware, C. (2020). Information Visualization: Perception for Design, Morgan Kaufmann.
98. ESRI (2022, January 05). ArcGIS 3D Analyst. Available online: https://www.esri.com/en-us/arcgis/products/arcgis-3d-analyst/overview.
99. Initial Validation of a Virtual Environment for Assessment of Memory Functioning: Virtual Reality Cognitive Performance Assessment Test10.1089/cpb.2007.9934 ↗
100. Hanus, A., Hoover, M., Lim, A., and Miller, J. (2019, January 23–27). A Collaborative Virtual Reality Escape Room with Passive Haptics. Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan.10.1109/VR.2019.8798241 ↗