An Approach for Selecting the Most Explanatory Features for Facial Expression Recognition
Source abstract
Abstract
The objective of this work is to analyze which features are most important in the recognition of facial expressions. To achieve this, we built a facial expression recognition system that learns from a controlled capture data set. The system uses different representations and combines them from a learned model. We studied the most important features by applying different feature extraction methods for facial expression representation, transforming each obtained representation into a sparse representation (SR) domain, and trained combination models to classify signals, using the extended Cohn–Kanade (CK+), BU-3DFE, and JAFFE data sets for validation. We compared 14 combination methods for 247 possible combinations of eight different feature spaces and obtained the most explanatory features for each facial expression. The results indicate that the LPQ (83%), HOG (82%), and RAW (82%) features are those features most able to improve the classification of expressions and that some features apply specifically to one expression (e.g., RAW for neutral, LPQ for angry and happy, LBP for disgust, and HOG for surprise).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Face and Expression Recognition · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Centro de Informática, Universidade Federal de Pernambuco (UFPE), Recife 50670-901, Brazil
Computer Graphics, Vision and Artificial Intelligence Group (UGIVIA), Department of Mathematics and Computer Science, University of the Balearic Islands, 07122 Palma, Spain
Laboratory of Artificial Intelligence Applications (LAIA@UIB), University of the Balearic Islands, 07122 Palma, Spain
Funding
MCIN/AEI/10.13039/501100011033
References (66)
Show source-provided references
1. Zhu, X., He, Z., Zhao, L., Dai, Z., and Yang, Q. (2022). A Cascade Attention Based Facial Expression Recognition Network by Fusing Multi-Scale Spatio-Temporal Features. Sensors, 22.10.3390/s22041350 ↗
2. Bonnard, J., Dapogny, A., Dhombres, F., and Bailly, K. (2022). Privileged Attribution Constrained Deep Networks for Facial Expression Recognition. arXiv.10.1109/ICPR56361.2022.9956496 ↗
3. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)10.1109/ACCESS.2018.2870052 ↗
4. Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
5. Sickle-cell disease diagnosis support selecting the most appropriate machine learning method: Towards a general and interpretable approach for cell morphology analysis from microscopy images10.1016/j.compbiomed.2020.104027 ↗
6. A survey of methods for explaining black box models10.1145/3236009 ↗
7. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model10.1214/15-AOAS848 ↗
8. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why should I trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
9. A survey on feature selection methods10.1016/j.compeleceng.2013.11.024 ↗
10. Deep-learned faces of pain and emotions: Elucidating the differences of facial expressions with the help of explainable AI methods Tief erlernte Gesichter von Schmerz und Emotionen: Aufklärung der Unterschiede von Gesichtsausdrücken mithilfe erklärbarer K10.1515/teme-2019-0024 ↗
11. Gund, M., Bharadwaj, A.R., and Nwogu, I. (2021, January 10–15). Interpretable emotion classification using temporal convolutional models. Proceedings of the 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9412134 ↗
12. Burg, K.T. (2021). Explaining Dnn Based Facial Expression Classifications. [Bachelor’s Thesis, Utrecht University]. Volume 7.
13. M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis10.1109/TVCG.2021.3114794 ↗
14. Expression Analysis Based on Face Regions in Read-world Conditions10.1007/s11633-019-1176-9 ↗
15. Jung, H., Lee, S., Yim, J., Park, S., and Kim, J. (2015, January 7–13). Joint fine-tuning in deep neural networks for facial expression recognition. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.341 ↗
16. Deramgozin, M., Jovanovic, S., Rabah, H., and Ramzan, N. (2021, January 24–26). A Hybrid Explainable AI Framework Applied to Global and Local Facial Expression Recognition. Proceedings of the 2021 IEEE International Conference on Imaging Systems and Techniques (IST), Kaohsiung, Taiwan.10.1109/IST50367.2021.9651357 ↗
17. Ying, Z.-L., Wang, Z.-W., and Huang, M.-W. (2010). Facial expression recognition based on fusion of sparse representation. Advanced Intelligent Computing Theories and Applications. With Aspects of Artificial Intelligence, Springer.10.1007/978-3-642-14932-0_57 ↗
18. Li, L., Ying, Z., and Yang, T. (2014, January 19–23). Facial expression recognition by fusion of gabor texture features and local phase quantization. Proceedings of the 2014 12th International Conference on Signal Processing (ICSP), Hangzhou, China.10.1109/ICOSP.2014.7015300 ↗
19. Accurate and robust facial expressions recognition by fusing multiple sparse representation based classifiers10.1016/j.neucom.2014.03.073 ↗
20. On combining classifiers10.1109/34.667881 ↗
21. Ptucha, R., and Savakis, A. (October, January 30). Fusion of static and temporal predictors for unconstrained facial expression recognition. Proceedings of the 2012 19th IEEE International Conference on Image Processing (ICIP), Orlando, FL, USA.10.1109/ICIP.2012.6467430 ↗
22. Automatic facial expression recognition based on spatiotemporal descriptors10.1016/j.patrec.2012.03.006 ↗
23. Real-time 2D+ 3D facial action and expression recognition10.1016/j.patcog.2009.12.009 ↗
24. Sparse coding with an overcomplete basis set: A strategy employed by V1?10.1016/S0042-6989(97)00169-7 ↗
25. Emergence of simple-cell receptive field properties by learning a sparse code for natural images10.1038/381607a0 ↗
26. Robust face recognition via sparse representation10.1109/TPAMI.2008.79 ↗
27. Weifeng, L., Caifeng, S., and Yanjiang, W. (2012, January 21–25). Facial expression analysis using a sparse representation based space model. Proceedings of the 2012 IEEE 11th International Conference on Signal Processing (ICSP), Beijing, China.
28. Zhen, W., and Zilu, Y. (2012, January 29–31). Facial expression recognition based on local phase quantization and sparse representation. Proceedings of the 2012 Eighth International Conference on Natural Computation (ICNC), Chongqing, China.
29. Mollahosseini, A., Chan, D., and Mahoor, M.H. (2016, January 7–10). Going deeper in facial expression recognition using deep neural networks. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477450 ↗
30. Zhang, S., Li, L., and Zhao, Z. (2012, January 21–25). Facial expression recognition based on Gabor wavelets and sparse representation. Proceedings of the 2012 IEEE 11th International Conference on Signal Processing (ICSP), Beijing, China.10.1109/ICoSP.2012.6491706 ↗
31. Manifold based sparse representation for facial understanding in natural images10.1016/j.imavis.2013.03.003 ↗
32. Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., and Metaxas, D.N. (2012, January 16–21). Learning active facial patches for expression analysis. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.
33. Baltrušaitis, T., Robinson, P., and Morency, L.-P. (2016, January 7–10). OpenFace: An open source facial behavior analysis toolkit. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Lake Placid, NY, USA.10.1109/WACV.2016.7477553 ↗
34. Automatic Facial Expression Recognition Using Features of Salient Facial Patches10.1109/TAFFC.2014.2386334 ↗
35. Parkhi, O.M., Vedaldi, A., and Zisserman, A. (2015, January 7–10). Deep Face Recognition. Proceedings of the BMVC, Swansea, UK.10.5244/C.29.41 ↗
36. Liu, P., Han, S., Meng, Z., and Tong, Y. (2014, January 23–28). Facial expression recognition via a boosted deep belief network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.233 ↗
37. A novel SRC fusion method using hierarchical multi-scale LBP and greedy search strategy10.1016/j.neucom.2014.10.041 ↗
38. Huang, M.-W., Wang, Z., and Ying, Z.-L. (2010, January 16–18). A new method for facial expression recognition based on sparse representation plus LBP. Proceedings of the 2010 3rd International Congress on Image and Signal Processing (CISP), Yantai, China.10.1109/CISP.2010.5647898 ↗
39. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of oriented gradients for human detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR 2005, San Diego, CA, USA.
40. Ojansivu, V., and Heikkilä, J. (2008). Blur Insensitive Texture Classification Using Local Phase Quantization. Image and Signal Processing, Springer.10.1007/978-3-540-69905-7_27 ↗
41. Visual classification with multitask joint sparse representation10.1109/TIP.2012.2205006 ↗
42. Yin, L., Chen, X., Sun, Y., Worm, T., and Reale, M. (2008, January 17–19). A high-resolution 3d dynamic facial expression database. Proceedings of the 2008 8th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2008, Amsterdam, The Netherlands.10.1109/AFGR.2008.4813324 ↗
43. Yin, L., Wei, X., Sun, Y., Wang, J., and Rosato, M.J. (2006, January 10–12). A 3D facial expression database for facial behavior research. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, FGR 2006, Southampton, UK.
44. Liu, M., Li, S., Shan, S., Wang, R., and Chen, X. (2014, January 1–5). Deeply learning deformable facial action parts model for dynamic expression analysis. Proceedings of the Asian Conference on Computer Vision, Singapore.10.1007/978-3-319-16817-3_10 ↗
45. Marrero Fernandez, P.D., Guerrero Pena, F.A., Ing Ren, T., and Cunha, A. (2019). FERAtt: Facial Expression Recognition with Attention Net. arXiv.10.1109/CVPRW.2019.00112 ↗
46. Manresa-Yee, C., and Ramis, S. (2021). Assessing Gender Bias in Predictive Algorithms Using EXplainable AI, Association for Computing Machinery.10.1145/3471391.3471420 ↗
47. Wang, Z., Qinami, K., Karakozis, I.C., Genova, K., Nair, P., Hata, K., and Russakovsky, O. (2020, January 13–19). Towards fairness in visual recognition: Effective strategies for bias mitigation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00894 ↗
48. Xu, T., White, J., Kalkan, S., and Gunes, H. (2020, January 23–28). Investigating Bias and Fairness in Facial Expression Recognition. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-65414-6_35 ↗
49. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
50. For most large underdetermined systems of linear equations the minimal ?1-norm solution is also the sparsest solution10.1002/cpa.20132 ↗
51. Decision templates for multiple classifier fusion: An experimental comparison10.1016/S0031-3203(99)00223-X ↗
52. Methods for combining experts’ probability assessments10.1162/neco.1995.7.5.867 ↗
53. Hierarchical mixtures of experts and the EM algorithm10.1162/neco.1994.6.2.181 ↗
54. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
55. Lyons, M., Akamatsu, S., Kamachi, M., and Gyoba, J. (1998, January 14–16). Coding facial expressions with gabor wavelets. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan.
56. A survey of affect recognition methods: Audio, visual, and spontaneous expressions10.1109/TPAMI.2008.52 ↗
57. Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., and Pietikäinen, M. (2015). Reading Hidden Emotions: Spontaneous Micro-expression Spotting and Recognition. arXiv.
58. Robust automatic facial expression detection method based on sparse representation plus LBP map10.1016/j.ijleo.2013.05.076 ↗
59. A weighted voting framework for classifiers ensembles10.1007/s10115-012-0586-6 ↗
60. Fusion of feature sets and classifiers for facial expression recognition10.1016/j.eswa.2012.07.074 ↗
61. Statistical comparisons of classifiers over multiple data sets
62. Spatiotemporal local monogenic binary patterns for facial expression recognition10.1109/LSP.2012.2188890 ↗
63. Simultaneous facial feature tracking and facial expression recognition10.1109/TIP.2013.2253477 ↗
64. Intra-Class Variation Reduction Using Training Expression Images for Sparse Representation Based Facial Expression Recognition10.1109/TAFFC.2014.2346515 ↗
65. Ramis, S., Buades, J.M., Perales, F.J., and Manresa-Yee, C. (2022). A Novel Approach to Cross dataset studies in Facial Expression Recognition. Multimed. Tools Appl.10.1007/s11042-022-13117-2 ↗
66. Dynamics of facial expression extracted automatically from video10.1016/j.imavis.2005.09.011 ↗