Gaze Estimation Based on a Multi-Stream Adaptive Feature Fusion Network
Source abstract
Abstract
Recently, with the widespread application of deep learning networks, appearance-based gaze estimation has made breakthrough progress. However, most methods focus on feature extraction from the facial region while neglecting the critical role of the eye region in gaze estimation, leading to insufficient eye detail representation. To address this issue, this paper proposes a multi-stream multi-input network architecture (MSMI-Net) based on appearance. The model consists of two independent streams designed to extract high-dimensional eye features and low-dimensional features, integrating both eye and facial information. A parallel channel and spatial attention mechanism is employed to fuse low-dimensional eye and facial features, while an adaptive weight adjustment mechanism (AWAM) dynamically determines the contribution ratio of eye and facial features. The concatenated high-dimensional and fused low-dimensional features are processed through fully connected layers to predict the final gaze direction. Extensive experiments on the EYEDIAP, MPIIFaceGaze, and Gaze360 datasets validate the superiority of the proposed method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Gaze Tracking and Assistive Technology · Hand Gesture Recognition Systems · Advanced Computing and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Artificial Intelligence, Nanjing University of Information Science and Technology, Nanjing 210044, China
School of Computer Science, Nanjing University of Information Science and Technology, Nanjing 210044, China
State Key Laboratory of Precision Blasting, Jianghan University, Wuhan 430056, China
Funding
National Natural Science Foundation of China
References (52)
Show source-provided references
1. Rajanna, V., and Hammond, T. (2018). A gaze-assisted multimodal approach to rich and accessible human-computer interaction. arXiv.
2. Andrist, S., Tan, X.Z., Gleicher, M., and Mutlu, B. (2014, January 3–6). Conversational gaze aversion for humanlike robots. Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction, Bielefeld, Germany.10.1145/2559636.2559666 ↗
3. Eye tracking in virtual reality10.16910/jemr.12.1.3 ↗
4. Patney, A., Kim, J., Salvi, M., Kaplanyan, A., Wyman, C., Benty, N., Lefohn, A., and Luebke, D. (2016, January 24–28). Perceptually-based foveated virtual reality. Proceedings of the ACM SIGGRAPH 2016 Emerging Technologies, Anaheim, CA, USA.10.1145/2929464.2929472 ↗
5. Evaluation of gaze tracking calibration for longitudinal biomedical imaging studies10.1109/TCYB.2018.2866274 ↗
6. Eye movements in reading and information processing: 20 years of research10.1037/0033-2909.124.3.372 ↗
7. Dynamics of driver’s gaze: Explorations in behavior modeling and maneuver prediction10.1109/TIV.2018.2804160 ↗
8. Zhu, Z., and Ji, Q. (2005, January 20–26). Eye gaze tracking under natural head movements. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
9. Wang, K., and Ji, Q. (2017, January 22–29). Real time eye gaze tracking with 3d deformable eye-face model. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.114 ↗
10. Kaur, H., Jindal, S., and Manduchi, R. (2022, January 7–11). Rethinking model-based gaze estimation. Proceedings of the ACM on Computer Graphics and Interactive Techniques, Vancouver, BC, Canada.
11. Learning gaze biases with head motion for head pose-free gaze estimation10.1016/j.imavis.2014.01.005 ↗
12. Appearance-based gaze estimation for ASD diagnosis10.1109/TCYB.2022.3165063 ↗
13. Elfares, M., Hu, Z., Reisert, P., Bulling, A., and Küsters, R. (2023, January 16). Federated learning for appearance-based gaze estimation in the wild. Proceedings of the Gaze Meets Machine Learning Workshop, PMLR, New Orleans, LA, USA.
14. Elfares, M., Reisert, P., Hu, Z., Tang, W., Küsters, R., and Bulling, A. (2024, January 11–16). PrivatEyes: Appearance-based Gaze Estimation Using Federated Secure Multi-Party Computation. Proceedings of the ACM on Human-Computer Interaction, O‘ahu, HI, USA.10.1145/3655606 ↗
15. 3D gaze estimation without explicit personal calibration10.1016/j.patcog.2018.01.031 ↗
16. Sugano, Y., Matsushita, Y., and Sato, Y. (2014, January 23–28). Learning-by-synthesis for appearance-based 3d gaze estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.235 ↗
17. Kendall, A., and Gal, Y. (2017). What uncertainties do we need in bayesian deep learning for computer vision?. Adv. Neural Inf. Process. Syst., 30.
18. Sugano, Y., Matsushita, Y., and Sato, Y. (2019, January 15–20). Generalizing eye tracking with bayesian adversarial learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.
19. Zhang, X., Sugano, Y., Fritz, M., and Bulling, A. (2015, January 7–12). Appearance-based gaze estimation in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299081 ↗
20. Cheng, Y., Lu, F., and Zhang, X. (2018, January 8–14). Appearance-based gaze estimation via evaluation-guided asymmetric regression. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_7 ↗
21. Cheng, Y., and Lu, F. (2022, January 21–25). Gaze estimation using transformer. Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR), Montreal, QC, Canada.10.1109/ICPR56361.2022.9956687 ↗
22. Shi, Y., Zhang, F., Yang, W., Wang, G., and Su, N. (2024, January 27–30). Agent-guided gaze estimation network by two-eye asymmetry exploration. Proceedings of the 2024 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.10.1109/ICIP51287.2024.10648029 ↗
23. Frequency-spatial interaction network for gaze estimation10.1016/j.displa.2024.102878 ↗
24. Combining head pose and eye location information for gaze estimation10.1109/TIP.2011.2162740 ↗
25. Nakazawa, A., and Nitschke, C. (2012). Point of gaze estimation through corneal surface reflection in an active illumination environment. Proceedings of the Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, 7–13 October 2012, Proceedings, Part II 12, Springer.
26. Sei, M., Utsumi, A., Yamazoe, H., and Lee, J. (June, January 30). Model-based deep gaze estimation using incrementally updated face-shape parame-ters. Proceedings of the 2023 Symposium on Eye Tracking Research and Applications, Tubingen, Germany.10.1145/3588015.3590135 ↗
27. Li, J., Yang, J., Liu, Y., Li, Z., Yang, G.Z., and Guo, Y. (2023, January 1–5). EasyGaze3D: Towards effective and flexible 3D gaze estimation from a single RGB camera. Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA.10.1109/IROS55552.2023.10342361 ↗
28. Shen, K., Li, Y., Guo, Z., Gao, J., and Wu, Y. (2024). Model-Based 3D Gaze Estimation Using a TOF Camera. Sensors, 24.10.3390/s24041070 ↗
29. Non-intrusive gaze tracking using artificial neural networks
30. Appearance-based gaze estimation using visual saliency10.1109/TPAMI.2012.101 ↗
31. Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., and Torralba, A. (2016, January 27–30). Eye tracking for everyone. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.239 ↗
32. Mpiigaze: Real-world dataset and deep appearance-based gaze estimation10.1109/TPAMI.2017.2778103 ↗
33. Cheng, Y., Huang, S., Wang, F., Qian, C., and Lu, F. (2020, January 7–12). A coarse-to-fine adaptive network for appearance-based gaze estimation. Proceedings of the AAAI Conference on Artificial Intelligence, Singapore.10.1609/aaai.v34i07.6636 ↗
34. Appearance-based gaze estimation with feature fusion of multi-level information elements
35. Ververas, E., Gkagkos, P., Deng, J., Doukas, M.C., Guo, J., and Zafeiriou, S. (October, January 29). 3DGazeNet: Generalizing 3D Gaze Estimation with Weak-Supervision from Synthetic Views. Proceedings of the European Conference on Computer Vision, Milan, Italy.
36. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., and Darrell, T. (2022, January 18–22). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52688.2022.01167 ↗
37. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
38. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
39. Funes Mora, K.A., Monay, F., and Odobez, J.M. (2014, January 26–28). Eyediap: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. Proceedings of the Symposium on Eye Tracking Research and Applications, Safety Harbor, FL, USA.10.1145/2578153.2578190 ↗
40. Kellnhofer, P., Recasens, A., Stent, S., Matusik, W., and Torralba, A. (November, January 27). Gaze360: Physically unconstrained gaze estimation in the wild. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
41. Chen, Z., and Shi, B.E. (2018, January 2–6). Appearance-based gaze estimation using dilated-convolutions. Proceedings of the Asian Conference on Computer Vision, Perth, Australia.10.1007/978-3-030-20876-9_20 ↗
42. Appearance-based gaze estimation with deep learning: A review and benchmark10.1109/TPAMI.2024.3393571 ↗
43. Kingma, D.P. (2014). Adam: A method for stochastic optimization. arXiv.
44. Bao, Y., Cheng, Y., Liu, Y., and Lu, F. (2021, January 10–15). Adaptive feature fusion network for gaze tracking in mobile tablets. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9412205 ↗
45. Gaze estimation by attention-induced hierarchical variational auto-encoder10.1109/TCYB.2023.3312392 ↗
46. Multistream gaze estimation with anatomical eye region isolation by synthetic to real transfer learning10.1109/TAI.2024.3366174 ↗
47. Fischer, T., Chang, H.J., and Demiris, Y. (2018, January 8–14). Rt-gene: Real-time eye gaze estimation in natural environments. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01249-6_21 ↗
48. Wang, H., Oh, J.O., Chang, H.J., Na, J.H., Tae, M., Zhang, Z., and Choi, S.I. (2023, January 18–22). Gazecaps: Gaze estimation with self-attention-routed capsules. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPRW59228.2023.00267 ↗
49. Qin, J., Zhang, X., and Sugano, Y. (2025). UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training. arXiv.
50. GazeSymCAT: A Symmetric Cross-Attention Transformer for Robust Gaze Estimation under Extreme Head Poses and Gaze Variations
51. Sinha, D., and El-Sharkawy, M. (2019, January 10–12). Thin mobilenet: An enhanced mobilenet architecture. Proceedings of the 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, USA.10.1109/UEMCON47517.2019.8993089 ↗
52. Hu, J., Shen, L., and Sun, G. (2018, January 18–22). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗