Context Transformer and Adaptive Method with Visual Transformer for Robust Facial Expression Recognition
Source abstract
Abstract
In real-world scenarios, the facial expression recognition task faces several challenges, including lighting variations, image noise, face occlusion, and other factors, which limit the performance of existing models in dealing with complex situations. To cope with these problems, we introduce the CoT module between the CNN and ViT frameworks, which improves the ability to perceive subtle differences by learning the correlations between local area features at a fine-grained level, helping to maintain the consistency between the local area features and the global expression, and making the model more adaptable to complex lighting conditions. Meanwhile, we adopt an adaptive learning method to effectively eliminate the interference of noise and occlusion by dynamically adjusting the parameters of the Transformer Encoder’s self-attention weight matrix. Experiments demonstrate the accuracy of our CoT_AdaViT model in the Oulu-CASIA dataset as (NIR: 87.94%, VL: strong: 89.47%, weak: 84.76%, dark: 82.28%). As well as, CK+, RAF-DB, and FERPlus datasets achieved 99.20%, 91.07%, and 90.57% recognition results, which achieved excellent performance and verified that the model has strong recognition accuracy and robustness in complex scenes.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face and Expression Recognition · Emotion and Mood Recognition · Face recognition and analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Dalian University of Technology, Dalian 116000, China
Neusoft Reach Automotive Technology (Dalian) Co., Ltd., Dalian 116000, China
References (55)
Show source-provided references
1. The early development of emotion recognition in autistic children: Decoding basic emotions from facial expressions and emotion-provoking situations
2. Munsif, M., Ullah, M., Ahmad, B., Sajjad, M., and Cheikh, F.A. (2022). Monitoring Neurological Disorder Patients via Deep Learning Based Facial Expressions Analysis, Springer.10.1007/978-3-031-08341-9_33 ↗
3. Kabir, M.R., Dewan, M.A.A., and Lin, F. (2023, January 24–26). Lightweight model for emotion detection from facial expression in online learning. Proceedings of the 2023 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), Regina, SK, Canada.10.1109/CCECE58730.2023.10288951 ↗
4. The Effect of Synchrony of Happiness on Facial Expression of Negative Emotion When Lying
5. An automated hyperparameter tuned deep learning model enabled facial emotion recognition for autonomous vehicle drivers10.1016/j.imavis.2023.104659 ↗
6. Hijji, M., Yar, H., Ullah, F.U.M., Alwakeel, M.M., Harrabi, R., Aradah, F., Cheikh, F.A., Muhammad, K., and Sajjad, M. (2023). FADS: An Intelligent Fatigue and Age Detection System. Mathematics, 11.10.3390/math11051174 ↗
7. Minaee, S., Minaei, M., and Abdolrashidi, A.J.S. (2021). Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors, 21.10.3390/s21093046 ↗
8. Shahzad, H., Bhatti, S.M., Jaffar, A., Akram, S., Alhajlah, M., and Mahmood, A. (2023). Hybrid Facial Emotion Recognition Using CNN-Based Features. Appl. Sci., 13.10.3390/app13095572 ↗
9. Region attention networks for pose and occlusion robust facial expression recognition10.1109/TIP.2019.2956143 ↗
10. Zhou, J., Zhang, X., and Liu, Y. (2020, January 1–4). Learning the connectivity: Situational graph convolution network for facial expression recognition. Proceedings of the 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), Macau, China.10.1109/VCIP49819.2020.9301773 ↗
11. Xu, C., Du, Y., Wang, J., Zheng, W., Li, T., and Yuan, Z. (2023). A joint hierarchical cross-attention graph convolutional network for multi-modal facial expression recognition. Comput. Intell.10.1111/coin.12607 ↗
12. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
13. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
14. Contextual transformer networks for visual recognition10.1109/TPAMI.2022.3164083 ↗
15. Facial expression recognition from near-infrared videos10.1016/j.imavis.2011.07.002 ↗
16. Jung, H., Lee, S., Yim, J., Park, S., and Kim, J. (2015, January 7–13). Joint fine-tuning in deep neural networks for facial expression recognition. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.341 ↗
17. Wu, Z., Chen, T., Chen, Y., Zhang, Z., and Liu, G.J.A.S. (2017). NIRExpNet: Three-stream 3D convolutional neural network for near infrared facial expression recognition. Appl. Sci., 7.10.3390/app7111184 ↗
18. Chen, Y., Zhang, Z.H., Zhong, L., Chen, T., Chen, J.X., and Yu, Y.D. (2019). Three-Stream Convolutional Neural Network with Squeeze-and-Excitation Block for Near-Infrared Facial Expression Recognition. Electronics, 8.10.3390/electronics8040385 ↗
19. Infrared facial expression recognition via Gaussian-based label distribution learning in the dark illumination environment for human emotion detection10.1016/j.neucom.2020.05.081 ↗
20. Recognition in the near infrared spectrum for face, gender and facial expressions10.1007/s11042-021-11728-9 ↗
21. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
22. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 1–26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
23. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 1–26). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
24. Tan, M., and Le, Q. (2019, January 10–15). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
25. Triple attention feature enhanced pyramid network for facial expression recognition10.3233/JIFS-222252 ↗
26. Expression recognition based on multi-regional coordinate attention residuals10.1109/ACCESS.2023.3285781 ↗
27. Facial expression recognition with grid-wise attention and visual transformer10.1016/j.ins.2021.08.043 ↗
28. Zhang, Y., Wang, C., Ling, X., and Deng, W. (2022, January 23–27). Learn from all: Erasing attention consistency for noisy label facial expression recognition. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19809-0_24 ↗
29. Emotion recognition from occluded facial images using deep ensemble model
30. Gómez-Sirvent, J.L., López de la Rosa, F., López, M.T., and Fernández-Caballero, A. (2023). Facial Expression Recognition in the Wild for Low-Resolution Images Using Voting Residual Network. Electronics, 12.10.3390/electronics12183837 ↗
31. CFNet: Facial expression recognition via constraint fusion under multi-task joint learning network10.1016/j.asoc.2023.110312 ↗
32. Naveen, P. (2023). Occlusion-aware facial expression recognition: A deep learning approach. Multimed. Tools Appl., 1–27.10.1007/s11042-023-17013-1 ↗
33. Efficient neural architecture search for emotion recognition10.1016/j.eswa.2023.119957 ↗
34. Bobojanov, S., Kim, B.M., Arabboev, M., and Begmatov, S.J.A.S. (2023). Comparative Analysis of Vision Transformer Models for Facial Emotion Recognition Using Augmented Balanced Datasets. Appl. Sci., 13.10.3390/app132212271 ↗
35. Aouayeb, M., Hamidouche, W., Soladie, C., Kpalma, K., and Seguier, R. (2021). Learning vision transformer with squeeze and excitation for facial expression recognition. arXiv.
36. Li, H., Sui, M., Zhao, F., Zha, Z., and Wu, F. (2021). MVT: Mask vision transformer for facial expression recognition in the wild. arXiv.
37. Facial expression recognition with visual transformers and attentional selective fusion10.1109/TAFFC.2021.3122146 ↗
38. Xue, F., Wang, Q., and Guo, G. (2021, January 11–17). Transfer: Learning relation-aware facial expression representations with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00358 ↗
39. Vision transformer with attentive pooling for robust facial expression recognition10.1109/TAFFC.2022.3226473 ↗
40. Yao, H., Yang, X., Chen, D., Wang, Z., and Tian, Y. (2023). Facial Expression Recognition Based on Fine-Tuned Channel–Spatial Attention Transformer. Sensors, 23.10.3390/s23156799 ↗
41. Jin, Z., Zhang, X., Wang, J., Xu, X., and Xiao, J. (2023). Fine-Grained Facial Expression Recognition in Multiple Smiles. Electronics, 12.10.3390/electronics12051089 ↗
42. A Robust Driver Emotion Recognition Method Based on High-Purity Feature Separation10.1109/TITS.2023.3304128 ↗
43. Facial expression recognition based on deep evolutional spatial-temporal networks10.1109/TIP.2017.2689999 ↗
44. Hinet: Hybrid inherited feature learning network for facial expression recognition10.1109/LOCS.2019.2927959 ↗
45. Ruan, D., Yan, Y., Lai, S., Chai, Z., Shen, C., and Wang, H. (2021, January 20–25). Feature decomposition and reconstruction learning for effective facial expression recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00757 ↗
46. Arabian, H., Battistel, A., Chase, J.G., and Moeller, K. (2023). Attention-Guided Network Model for Image-Based Emotion Recognition. Appl. Sci., 13.10.3390/app131810179 ↗
47. Wen, Z., Lin, W., Wang, T., and Xu, G. (2023). Distract your attention: Multi-head cross attention network for facial expression recognition. Biomimetics, 8.10.3390/biomimetics8020199 ↗
48. Real emotion seeker: Recalibrating annotation for facial expression recognition10.1007/s00530-022-00986-8 ↗
49. Jiang, Q., Peng, X., Chen, H., and Guo, Y. (2022, January 25–27). Facial expression recognition based on residual network. Proceedings of the 2022 41st Chinese Control Conference (CCC), Hefei, China.10.23919/CCC55666.2022.9902268 ↗
50. Role of zoning in facial expression using deep learning10.1109/ACCESS.2023.3243850 ↗
51. Pourmirzaei, M., Montazer, G.A., and Esmaili, F. (2021). Using self-supervised auxiliary tasks to improve fine-grained facial representation. arXiv.
52. Emotion Recognition from Facial Expression Using Hybrid cnn–lstm Network10.1142/S0218001423560086 ↗
53. NAGNet: A novel framework for real-time students’ sentiment analysis in the wisdom classroom10.1002/cpe.7727 ↗
54. Efficient facial expression recognition with representation reinforcement network and transfer self-training for human–machine interaction10.1109/TII.2022.3233650 ↗
55. Fine-tuning swin transformer and multiple weights optimality-seeking for facial expression recognition10.1109/ACCESS.2023.3237817 ↗