Appearance-Based Gaze Estimation as a Benchmark for Eye Image Data Generation Methods
Source abstract
Abstract
Data augmentation is commonly utilized to increase the size and diversity of training sets for deep learning tasks. In this study, we propose a novel application of an existing image generation approach in the domain of realistic eye images that leverages data collected from 40 subjects. This hybrid method combines the benefits of precise control over the image content provided by 3D rendering, while introducing the previously lacking photorealism and diversity into synthetic images through neural style transfer. We prove its general efficacy as a data augmentation tool for appearance-based gaze estimation when generated data are mixed with a sparse train set of real images. It improved the results for 39 out of 40 subjects, with an 11.22% mean and a 19.75% maximum decrease in gaze estimation error, achieving similar metrics for train and held-out subjects. We release our data repository of eye images with gaze labels used in this work for public access.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Gaze Tracking and Assistive Technology · Video Surveillance and Tracking Methods · Retinal Imaging and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Texas State University, 601 University Drive, San Marcos, TX 78666, USA
References (68)
Show source-provided references
1. Salvucci, D.D., and Anderson, J.R. (2000, January 1–6). Intelligent gaze-added interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, The Hague, The Netherlands.10.1145/332040.332444 ↗
2. Møllenbach, E., Hansen, J.P., and Lillholm, M. (2013). Eye movements in gaze interaction. J. Eye Mov. Res., 6.10.16910/jemr.6.2.1 ↗
3. Fares, R., Fang, S., and Komogortsev, O. (May, January 27). Can we beat the mouse with MAGIC?. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France.
4. Foveated 3D graphics10.1145/2366145.2366183 ↗
5. Towards foveated rendering for gaze-tracked virtual reality10.1145/2980179.2980246 ↗
6. Rigas, I., Friedman, L., and Komogortsev, O. (2018). Study of an extensive set of eye movement features: Extraction methods and statistical analysis. J. Eye Mov. Res., 11.10.16910/jemr.11.1.3 ↗
7. Lohr, D., Proulx, M.J., and Komogortsev, O. (2024). Establishing a Baseline for Gaze-driven Authentication Performance in VR: A Breadth-First Investigation on a Very Large Dataset. arXiv.10.1109/IJCB62174.2024.10744483 ↗
8. Komogortsev, O.V., and Holland, C.D. (2014). The application of eye movement biometrics in the automated detection of mild traumatic brain injury. CHI’14 Extended Abstracts on Human Factors in Computing Systems, Association for Computing Machinery.10.1145/2559206.2581150 ↗
9. Eye tracking, attention, and schizotypal symptoms in nonpsychotic relatives of patients with schizophrenia10.1001/archpsyc.1997.01830140081014 ↗
10. Quantitative assessment of smooth pursuit gain and catch-up saccades in schizophrenia and affective disorders10.1016/0006-3223(91)90248-K ↗
11. Eye tracking algorithms, techniques, tools, and applications with an emphasis on machine learning and Internet of Things technologies10.1016/j.eswa.2020.114037 ↗
12. In the eye of the beholder: A survey of models for eyes and gaze10.1109/TPAMI.2009.30 ↗
13. General theory of remote gaze estimation using the pupil center and corneal reflections10.1109/TBME.2005.863952 ↗
14. Mpiigaze: Real-world dataset and deep appearance-based gaze estimation10.1109/TPAMI.2017.2778103 ↗
15. Kellnhofer, P., Recasens, A., Stent, S., Matusik, W., and Torralba, A. (November, January 27). Gaze360: Physically unconstrained gaze estimation in the wild. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
16. Cheng, Y., and Lu, F. (2022, January 21–25). Gaze estimation using transformer. Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR), Montreal, QC, Canada.10.1109/ICPR56361.2022.9956687 ↗
17. Cheng, Y., Wang, H., Bao, Y., and Lu, F. (2024). Appearance-based gaze estimation with deep learning: A review and benchmark. IEEE Trans. Pattern Anal. Mach. Intell., 1–20.10.1109/TPAMI.2024.3393571 ↗
18. Perez, L., and Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv.
19. Deep convolutional neural networks and data augmentation for environmental sound classification10.1109/LSP.2017.2657381 ↗
20. A survey on Image Data Augmentation for Deep Learning10.1186/s40537-019-0197-0 ↗
21. Katrychuk, D., Griffith, H.K., and Komogortsev, O.V. (2019, January 25–28). Power-efficient and shift-robust eye-tracking sensor for portable VR headsets. Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, Denver, CO, USA.10.1145/3314111.3319821 ↗
22. Kim, J., Stengel, M., Majercik, A., De Mello, S., Dunn, D., Laine, S., McGuire, M., and Luebke, D. (2019, January 4–9). Nvgaze: An anatomically-informed dataset for low-latency, near-eye gaze estimation. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290605.3300780 ↗
23. Świrski, L., and Dodgson, N. (2014, January 26–28). Rendering synthetic ground truth images for eye tracker evaluation. Proceedings of the Symposium on Eye Tracking Research and Applications, Safety Harbor, FL, USA.10.1145/2578153.2578188 ↗
24. Choi, Y., Uh, Y., Yoo, J., and Ha, J.W. (2020, January 13–19). Stargan v2: Diverse image synthesis for multiple domains. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00821 ↗
25. Kaur, H., and Manduchi, R. (2020, January 1–5). EyeGAN: Gaze–Preserving, Mask–Mediated Eye Image Synthesis. Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass, CO, USA.10.1109/WACV45572.2020.9093433 ↗
26. Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., and Webb, R. (2017, January 21–26). Learning from simulated and unsupervised images through adversarial training. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.241 ↗
27. Wood, E., Baltrušaitis, T., Morency, L.P., Robinson, P., and Bulling, A. (2016, January 14–17). Learning an appearance-based gaze estimator from one million synthesised images. Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, Charleston, SC, USA.10.1145/2857491.2857492 ↗
28. Nair, N., Chaudhary, A.K., Kothari, R.S., Diaz, G.J., Pelz, J.B., and Bailey, R. (2020, January 2–5). RIT-Eyes: Realistically rendered eye images for eye-tracking applications. Proceedings of the Symposium on Eye Tracking Research and Applications, Stuttgart, Germany.10.1145/3379157.3391990 ↗
29. Agtzidis, I., Startsev, M., and Dorr, M. (2019). A Ground-Truth Data Set and a Classification Algorithm for Eye Movements in 360-degree Videos. arXiv.
30. Generative adversarial nets
31. Brock, A., Donahue, J., and Simonyan, K. (2018). Large scale GAN training for high fidelity natural image synthesis. arXiv.
32. Alias-free generative adversarial networks
33. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis
34. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–27). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
35. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
36. Mathur, A., Isopoussu, A., Kawsar, F., Berthouze, N., and Lane, N.D. (2019, January 16–18). Mic2mic: Using cycle-consistent generative adversarial networks to overcome microphone variability in speech systems. Proceedings of the 18th International Conference on Information Processing in Sensor Networks, Montreal, QC, Canada.10.1145/3302506.3310398 ↗
37. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., and Paul Smolley, S. (2017, January 22–29). Least squares generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.304 ↗
38. Improved training of wasserstein gans
39. Kodali, N., Abernethy, J., Hays, J., and Kira, Z. (2017). On convergence and stability of gans. arXiv.
40. Are gans created equal? A large-scale study
41. Pros and cons of GAN evaluation measures: New developments10.1016/j.cviu.2021.103329 ↗
42. Denoising diffusion probabilistic models
43. Diffusion models beat gans on image synthesis
44. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. (2021). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv.
45. Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. (2023). Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv.
46. Improving image generation with better captions
47. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 19–20). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LO, USA.10.1109/CVPR52688.2022.01042 ↗
48. Patch diffusion: Faster and more data-efficient training of diffusion models
49. Antoniou, A., Storkey, A., and Edwards, H. (2017). Data Augmentation Generative Adversarial Networks. arXiv.
50. Sandfort, V., Yan, K., Pickhardt, P.J., and Summers, R.M. (2019). Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Sci. Rep., 9.10.1038/s41598-019-52737-x ↗
51. Sela, M., Xu, P., He, J., Navalpakkam, V., and Lagun, D. (2017). Gazegan-unpaired adversarial image generation for gaze estimation. arXiv.
52. Gatys, L.A., Ecker, A.S., and Bethge, M. (2016, January 27–30). Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.265 ↗
53. Buhler, M., Park, S., De Mello, S., Zhang, X., and Hilliges, O. (2019, January 27–28). Content-consistent generation of realistic eyes with style. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Republic of Korea.10.1109/ICCVW48693.2019.9130178 ↗
54. Johnson, J., Alahi, A., and Fei-Fei, L. (2016, January 11–14). Perceptual losses for real-time style transfer and super-resolution. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_43 ↗
55. Park, T., Liu, M.Y., Wang, T.C., and Zhu, J.Y. (2019, January 15–20). Semantic image synthesis with spatially-adaptive normalization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00244 ↗
56. Huang, X., and Belongie, S. (2017, January 22–29). Arbitrary style transfer in real-time with adaptive instance normalization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.167 ↗
57. Kaur, H., and Manduchi, R. (2021, January 5–9). Subject guided eye image synthesis with application to gaze redirection. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual.10.1109/WACV48630.2021.00006 ↗
58. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., and Choo, J. (2018, January 18–23). Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00916 ↗
59. Katrychuk, D., Griffith, H., and Komogortsev, O. (2020, January 2–5). A Calibration Framework for Photosensor-based Eye-Tracking System. Proceedings of the Symposium on Eye Tracking Research and Applications, Stuttgart, Germany.10.1145/3379156.3391370 ↗
60. Abdulin, E., Friedman, L., and Komogortsev, O. (2019). Custom video-oculography device and its application to fourth purkinje image detection during saccades. arXiv.
61. Chaudhary, A.K., Kothari, R., Acharya, M., Dangi, S., Nair, N., Bailey, R., Kanan, C., Diaz, G., and Pelz, J.B. (2019, January 27–28). Ritnet: Real-time semantic segmentation of the eye for gaze tracking. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Republic of Korea.10.1109/ICCVW.2019.00568 ↗
62. (2023, June 28). Facebook. 2019 OpenEDS Challenge for Eye Image Segmentation. Available online: https://research.facebook.com/openeds-challenge/.
63. Gans trained by a two time-scale update rule converge to a local nash equilibrium
64. lmerTest package: Tests in linear mixed effects models10.18637/jss.v082.i13 ↗
65. Random effects structure for confirmatory hypothesis testing: Keep it maximal10.1016/j.jml.2012.11.001 ↗
66. A kenward-roger approximation and parametric bootstrap methods for tests in linear mixed models—The R package pbkrtest10.18637/jss.v059.i09 ↗
67. Package ‘emmeans’
68. EyeNeRF: A hybrid representation for photorealistic synthesis, animation and relighting of human eyes