HSAW: A Half-Face Self-Attention Weighted Approach for Facial Expression Recognition
Source abstract
Abstract
Facial expression recognition plays an increasingly important role in daily life, and it is used in several areas of human–computer interaction, such as robotics, assisted driving, and intelligent tutoring systems. However, the current mainstream methods are based on the whole face, and do not consider the existence of expression asymmetry between the left and right half-face. Hence, the accuracy of facial expression recognition needs to be improved. In this paper, we propose a half-face self-attention weighted approach called HSAW. Using statistical analysis and computer vision techniques, we found that the left half-face contains richer expression features than the right half-face. Specifically, we employed a self-attention mechanism to assign different weights to the left and right halves of the face. These weights are combined with convolutional neural network features for improved facial expression recognition. Furthermore, to attack the presence of uncertain categories in the dataset, we introduce adaptive re-labeling module, which can improve the recognition accuracy. Extensive experiments conducted on the FER2013 and RAF datasets have verified the effectiveness of the proposed method, which utilizes fewer parameters.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Face and Expression Recognition · Gaze Tracking and Assistive Technology
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Jiangsu University of Science and Technology, Zhenjiang 212003, China
Funding
National Natural Science Foundation of China
References (35)
Show source-provided references
1. Martini, M., Bufalari, I., Stazi, M.A., and Aglioti, S.M. (2015). Is that me or my twin? Lack of self-face recognition advantage in identical twins. PLoS ONE, 10.10.1371/journal.pone.0120900 ↗
2. Asymmetry of facial expression in spontaneous emotion10.1016/S0010-9452(84)80041-6 ↗
3. Dynamic multi-channel metric network for joint pose-aware and identity-invariant facial expression recognition10.1016/j.ins.2021.07.034 ↗
4. Region attention networks for pose and occlusion robust facial expression recognition10.1109/TIP.2019.2956143 ↗
5. Active appearance models10.1109/34.927467 ↗
6. A comparative study of texture measures with classification based on featured distributions10.1016/0031-3203(95)00067-4 ↗
7. Support vector machines in remote sensing: A review10.1016/j.isprsjprs.2010.11.001 ↗
8. Facial expression recognition based on deep evolutional spatial-temporal networks10.1109/TIP.2017.2689999 ↗
9. Mehta, D., Siddiqui, M.F.H., and Javaid, A.Y. (2019). Recognition of emotion intensities using machine learning algorithms: A comparative study. Sensors, 19.10.3390/s19081897 ↗
10. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
11. Facial expression recognition method based on improved VGG convolutional neural network10.1142/S0218001420560030 ↗
12. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
13. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst., 25.
14. A visual attention based ROI detection method for facial expression recognition10.1016/j.neucom.2018.03.034 ↗
15. Cai, J., Meng, Z., Khan, A.S., O’Reilly, J., Li, Z., Han, S., and Tong, Y. (2021, January 19–22). Identity-free facial expression recognition using conditional generative adversarial network. Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.10.1109/ICIP42928.2021.9506593 ↗
16. Cadieu, C.F., Hong, H., Yamins, D.L., Pinto, N., Ardila, D., Solomon, E.A., Majaj, N.J., and DiCarlo, J.J. (2014). Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS Comput. Biol., 10.10.1371/journal.pcbi.1003963 ↗
17. Baek, S., Song, M., Jang, J., Kim, G., and Paik, S.B. (2019). Spontaneous generation of face recognition in untrained deep neural networks. Biorxiv.10.1101/857466 ↗
18. Neural encoding and decoding with deep learning for dynamic natural vision10.1093/cercor/bhx268 ↗
19. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
20. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., and Wang, Z. (2016, January 27–30). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.207 ↗
21. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
22. Wang, K., Peng, X., Yang, J., Lu, S., and Qiao, Y. (2020, January 13–19). Suppressing uncertainties for large-scale facial expression recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00693 ↗
23. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.H. (2013, January 3–7). Challenges in representation learning: A report on three machine learning contests. Proceedings of the Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Republic of Korea. Proceedings, Part III 20.
24. Geometry guided pose-invariant facial expression recognition10.1109/TIP.2020.2972114 ↗
25. Zeng, G., Zhou, J., Jia, X., Xie, W., and Shen, L. (2018, January 15–19). Hand-crafted feature guided deep learning for facial expression recognition. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00068 ↗
26. Ming, Z., Chazalon, J., Luqman, M.M., Visani, M., and Burie, J.C. (2018, January 20–24). FaceLiveNet: End-to-end networks combining face verification with interactive facial expression-based liveness detection. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.10.1109/ICPR.2018.8545274 ↗
27. Recognizing facial expressions using a shallow convolutional neural network10.1109/ACCESS.2019.2921220 ↗
28. El Boudouri, Y., and Bohi, A. (2023, January 27–29). EmoNeXt: An Adapted ConvNeXt for Facial Emotion Recognition. Proceedings of the 2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP), Poitiers, France.10.1109/MMSP59012.2023.10337732 ↗
29. Pham, L., Vu, T.H., and Tran, T.A. (2021, January 10–15). Facial expression recognition using residual masking network. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9411919 ↗
30. Farzaneh, A.H., and Qi, X. (2021, January 5–9). Facial expression recognition in the wild via deep attentive center loss. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual.10.1109/WACV48630.2021.00245 ↗
31. Pyramid with super resolution for in-the-wild facial expression recognition10.1109/ACCESS.2020.3010018 ↗
32. Wen, Z., Lin, W., Wang, T., and Xu, G. (2023). Distract your attention: Multi-head cross attention network for facial expression recognition. Biomimetics, 8.10.3390/biomimetics8020199 ↗
33. Multi-relations aware network for in-the-wild facial expression recognition10.1109/TCSVT.2023.3234312 ↗
34. Zhang, S., Zhang, Y., Zhang, Y., Wang, Y., and Song, Z. (2023). A Dual-Direction Attention Mixed Feature Network for Facial Expression Recognition. Electronics, 12.10.3390/electronics12173595 ↗
35. Chen, Y., Li, J., Shan, S., Wang, M., and Hong, R. (2023). From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos. arXiv.