Viewpoint Robustness of Automated Facial Action Unit Detection Systems
Source abstract
Abstract
Automatic facial action detection is important, but no previous studies have evaluated pre-trained models on the accuracy of facial action detection as the angle of the face changes from frontal to profile. Using static facial images obtained at various angles (0°, 15°, 30°, and 45°), we investigated the performance of three automated facial action detection systems (FaceReader, OpenFace, and Py-feat). The overall performance was best for OpenFace, followed by FaceReader and Py-Feat. The performance of FaceReader significantly decreased at 45° compared to that at other angles, while the performance of Py-Feat did not differ among the four angles. The performance of OpenFace decreased as the target face turned sideways. Prediction accuracy and robustness to angle changes varied with the target facial components and action detection system.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Face recognition and analysis · Face Recognition and Perception
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Psychological Process Team, Guardian Robot Project, RIKEN, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan
Faculty of Arts, Kyoto University of the Arts, 2-116 Uryuyama Kitashirakawa, Sakyo, Kyoto 606-8271, Japan
Funding
Japan Science and Technology Agency
References (57)
Show source-provided references
1. The rise of affectivism10.1038/s41562-021-01130-8 ↗
2. Alameda-Pineda, X., Ricci, E., and Sebe, N. (2019). Affective facial computing: Generalizability across domains. Multimodal Behavior Analysis in the Wild, Academic Press.10.1016/B978-0-12-814601-9.00011-0 ↗
3. Li, S., and Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput.
4. Automatic analysis of facial actions: A survey10.1109/TAFFC.2017.2731763 ↗
5. A comprehensive survey on automatic facial action unit analysis10.1007/s00371-019-01707-5 ↗
6. Some pragmatic functions of conversational facial gestures10.1075/gest.00012.bav ↗
7. The emerging view of emotion as social information10.1111/j.1751-9004.2010.00262.x ↗
8. Ekman, P., Friesen, W.V., and Hager, J.C. (2002). Facial Action Coding System, Research Nexus eBook. [2nd ed.].
9. Mandak, M.K., and Awasthi, A. (2015). Automatic facial expression analysis. Understanding Facial Expressions in Communication, Springer.10.1007/978-81-322-1934-7 ↗
10. JÂA-net: Joint facial action unit detection and face alignment via adaptive attention10.1007/s11263-020-01378-z ↗
11. Adaptive 3D facial action intensity estimation and emotion recognition10.1016/j.eswa.2014.08.042 ↗
12. Improving speech related facial action unit recognition by audiovisual information fusion10.1109/TCYB.2018.2840090 ↗
13. Liu, Y., Zhang, X., Zhou, J., Li, X., Li, Y., Zhao, G., and Li, Y. (2021). Graph-based Facial Affect Analysis: A Review of Methods, Applications and Challenges. arXiv.
14. Are people happy when they smile?: Affective assessments based on automatic smile genuineness identification
15. Dupré, D., Krumhuber, E.G., Küster, D., and McKeown, G.J. (2020). A performance comparison of eight commercially available automatic classifiers for facial affect recognition. PLoS ONE, 15.10.1371/journal.pone.0231968 ↗
16. Baltrušaitis, T., Zadeh, A., Lim, Y.C., and Morency, L.P. (2018, January 15–19). OpenFace 2.0: Facial behavior analysis toolkit. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG), Xi’an, China.10.1109/FG.2018.00019 ↗
17. Crossing domains for AU coding: Perspectives, approaches, and measures10.1109/TBIOM.2020.2977225 ↗
18. Cheong, J.H., Xie, T., Byrne, S., and Chang, L.J. (2021). Py-Feat: Python Facial Expression Analysis Toolbox. arXiv.
19. Namba, S., Sato, W., Osumi, M., and Shimokawa, K. (2021). Assessing Automated Facial Action Unit Detection Systems for Analyzing Cross-Domain Facial Expression Databases. Sensors, 21.10.3390/s21124222 ↗
20. Surcinelli, P., Andrei, F., Montebarocci, O., and Grandi, S. (2021). Emotion Recognition of Facial Expressions Presented in Profile. Psychol. Rep., 00332941211018403.10.1177/00332941211018403 ↗
21. Face in profile view reduces perceived facial expression intensity: An eye-tracking study10.1016/j.actpsy.2014.12.001 ↗
22. Judgments of facial expressions of emotion in profile10.1037/a0024356 ↗
23. Information and viewpoint dependence in face recognition10.1016/S0010-0277(96)00785-8 ↗
24. Facial first impressions from another angle: How social judgements are influenced by changeable and invariant facial properties10.1111/bjop.12206 ↗
25. Looking at you or looking elsewhere: The influence of head orientation on the signal value of emotional facial expressions10.1007/s11031-007-9057-x ↗
26. Ekman, P., and Friesen, W.V. (1976). Pictures of Facial Affect, Consulting Psychologist.
27. Facial expressions of basic emotions in Japanese laypeople10.3389/fpsyg.2019.00259 ↗
28. Zadeh, A., Chong, L.Y., Baltrusaitis, T., and Morency, L.P. (2017, January 22–29). Convolutional experts constrained local model for 3d facial landmark detection. Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy.10.1109/ICCVW.2017.296 ↗
29. Savran, A., Alyüz, N., Dibeklio ğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., and Akarun, L. (2008, January 7–8). Bosphorus database for 3D face analysis. Proceedings of the European Workshop on Biometrics and Identity Management, Roskilde, Denmark.10.1007/978-3-540-89991-4_6 ↗
30. Bp4d-spontaneous: A high-resolution spontaneous 3D dynamic facial expression database10.1016/j.imavis.2014.06.002 ↗
31. DISFA: A spontaneous facial action intensity database10.1109/T-AFFC.2013.4 ↗
32. Valstar, M.F., Jiang, B., Mehu, M., Pantic, M., and Scherer, K. (2011, January 21–25). The first facial expression recognition and analysis challenge. Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA.10.1109/FG.2011.5771374 ↗
33. The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent10.1109/T-AFFC.2011.20 ↗
34. Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., and Matthews, I. (2011, January 21–25). Painful data: The UNBC-McMaster shoulder pain expression archive database. Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA.10.1109/FG.2011.5771462 ↗
35. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
36. Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., and Zafeiriou, S. (2019). Retinaface: Single-stage dense face localisation in the wild. arXiv.10.1109/CVPR42600.2020.00525 ↗
37. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
38. Kollias, D., Nicolaou, M.A., Kotsia, I., Zhao, G., and Zafeiriou, S. (2017, January 21–26). Recognition of affect in the wild using deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.247 ↗
39. Kollias, D., and Zafeiriou, S. (2018). Aff-wild2: Extending the Aff-wild database for affect recognition. arXiv.
40. Kollias, D., and Zafeiriou, S. (2018). A multi-task learning & generation framework: Valence–arousal, action units & primary expressions. arXiv.
41. Kollias, D., and Zafeiriou, S. (2019). Expression, affect, action unit recognition: Aff-wild2, multi-task learning and ArcFace. arXiv.
42. Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond10.1007/s11263-019-01158-4 ↗
43. Zafeiriou, S., Kollias, D., Nicolaou, M.A., Papaioannou, A., Zhao, G., and Kotsia, I. (2017, January 21–26). Aff-wild: Valence and arousal ‘n-the-Wild’ challenge. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.248 ↗
44. Jeni, L.A., Cohn, J.F., and De La Torre, F. (2015, January 2–5). Facing imbalanced data—Recommendations for the use of performance metrics. Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction, Washington, DC, USA.
45. Dowle, M., and Srinivasan, A. (2021, October 06). Data.table: Extension of ‘data.frame’. R Package, Version 1.14.2. Available online: https://CRAN.R-project.org/package=data.table.
46. Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.C., and Müller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform., 12.10.1186/1471-2105-12-77 ↗
47. Welcome to the Tidyverse10.21105/joss.01686 ↗
48. Zhang, N., Luo, J., and Gao, W. (2020, January 25–27). Research on Face Detection Technology Based on MTCNN. Proceedings of the 2020 International Conference on Computer Network, Electronic and Automation (ICCNEA), Xi’an, China.10.1109/ICCNEA50255.2020.00040 ↗
49. Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., and Li, S.Z. (2017, January 1–4). FaceBoxes: A CPU real-time face detector with high accuracy. Proceedings of the 2017 IEEE International Joint Conference on Biometrics (IJCB), Denver, CO, USA.10.1109/BTAS.2017.8272675 ↗
50. Zhao, K., Chu, W.S., and Zhang, H. (2016, January 27–30). Deep region and multi-label learning for facial action unit detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.369 ↗
51. Guo, X., Li, S., Yu, J., Zhang, J., Ma, J., Ma, L., Liu, W., and Ling, H. (2019). PFLD: A practical facial landmark detector. arXiv.
52. Chen, S., Liu, Y., Gao, X., and Han, Z. (2018, January 11–12). Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices. Proceedings of the Chinese Conference on Biometric Recognition, Urumchi, China.10.1007/978-3-319-97909-0_46 ↗
53. Garcia, R.V., Wandzik, L., Grabner, L., and Krueger, J. (2019, January 4–7). The harms of demographic bias in deep face recognition research. Proceedings of the 2019 International Conference on Biometrics (ICB), Crete, Greece.10.1109/ICB45273.2019.8987334 ↗
54. Evidence and a computational explanation of cultural differences in facial expression recognition10.1037/a0020019 ↗
55. Systematic Evaluation of Design Choices for Deep Facial Action Coding Across Pose10.3389/fcomp.2021.636094 ↗
56. Hassner, T., Harel, S., Paz, E., and Enbar, R. (2015, January 7–12). Effective face frontalization in unconstrained images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299058 ↗
57. Sagonas, C., Panagakis, Y., Zafeiriou, S., and Pantic, M. (2015, January 7–13). Robust statistical face frontalization. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.441 ↗