3D Approaches and Challenges in Facial Expression Recognition Algorithms—A Literature Review
Source abstract
Abstract
In recent years, facial expression analysis and recognition (FER) have emerged as an active research topic with applications in several different areas, including the human-computer interaction domain. Solutions based on 2D models are not entirely satisfactory for real-world applications, as they present some problems of pose variations and illumination related to the nature of the data. Thanks to technological development, 3D facial data, both still images and video sequences, have become increasingly used to improve the accuracy of FER systems. Despite the advance in 3D algorithms, these solutions still have some drawbacks that make pure three-dimensional techniques convenient only for a set of specific applications; a viable solution to overcome such limitations is adopting a multimodal 2D+3D analysis. In this paper, we analyze the limits and strengths of traditional and deep-learning FER techniques, intending to provide the research community an overview of the results obtained looking to the next future. Furthermore, we describe in detail the most used databases to address the problem of facial expressions and emotions, highlighting the results obtained by the various authors. The different techniques used are compared, and some conclusions are drawn concerning the best recognition rates achieved.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Face and Expression Recognition · Gaze Tracking and Assistive Technology
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Management and Production Engineering, Politecnico di Torino, 10129 Torino, Italy
References (100+)
Show source-provided references
1. Ko, C.B. (2018). A brief review of facial emotion recognition based on visual information. Sensors, 18.10.3390/s18020401 ↗
2. Multimodal 2D + 3D facial expression recognition with deep fusion convolutional neural network10.1109/TMM.2017.2713408 ↗
3. Bejaoui, H., Ghazouani, H., and Barhoumi, W. (2017, January 18–21). Fully automated facial expression recognition using 3D morphable model and mesh-local binary pattern. Proceedings of the International Conference on Advanced Concepts for Intelligent Vision Systems (ACIVIS), Antwerp, Belgium.10.1007/978-3-319-70353-4_4 ↗
4. Mishra, B., Fernandes, S.L., Abhishek, K., Alva, A., Shetty, C., Ajila, C.V., Shetty, D., Rao, H., and Shetty, P. (2015, January 26–27). Facial expression recognition using feature based techniques and model based techniques: A survey. Proceedings of the 2015 2nd International Conference on Electronics and Communication Systems (ICECS), Coimbatore, India.10.1109/ECS.2015.7124976 ↗
5. A survey on facial expression recognition in 3D video sequences10.1007/s11042-014-1869-6 ↗
6. Automatic analysis of facial affect: A survey of registration, representation, and recognition10.1109/TPAMI.2014.2366127 ↗
7. Survey on real-time facial expression recognition techniques10.1049/iet-bmt.2014.0104 ↗
8. Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., and Kakadiaris, I.A. (2011, January 21–23). 3D facial expression recognition: A perspective on promises and challenges. Proceedings of the Face and Gesture 2011, Santa Barbara, CA, USA.10.1109/FG.2011.5771466 ↗
9. Static and dynamic 3D facial expression recognition: A comprehensive survey10.1016/j.imavis.2012.06.005 ↗
10. Automatic analysis of facial expressions: The state of the art10.1109/34.895976 ↗
11. A survey of affect recognition methods: Audio, visual, and spontaneous expressions10.1109/TPAMI.2008.52 ↗
12. Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications10.1109/TPAMI.2016.2515606 ↗
13. Li, S., and Deng, W. (2018). Deep facial expression recognition: A survey. arXiv, Available online: https://arxiv.org/abs/1804.08348.
14. Revina, I.M., and Emmanuel, W.R.S. (2018). A survey on human face expression recognition techniques. J. King Saud Univ. Comput. Inf. Sci.
15. Constants across cultures in the face and emotion10.1037/h0030377 ↗
16. Ekman, P.E., and Davidson, R.J. (1994). The Nature of Emotion: Fundamental Questions, Oxford University Press.
17. Compound facial expressions of emotion10.1073/pnas.1322355111 ↗
18. Kawulok, M., Celebi, M.E., and Smolka, B. (2016). Advances, challenges, and opportunities in automatic facial expression recognition. Advances in Face Detection and Facial Image Analysis, Springer International Publishing.10.1007/978-3-319-25958-1 ↗
19. Kawulok, M., Celebi, E., and Smolka, B. (2016). Advances in Face Detection and Facial Image Analysis, Springer.10.1007/978-3-319-25958-1 ↗
20. Ekman, P., and Friesen, W.V. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press.10.1037/t27734-000 ↗
21. Recognizing action units for facial expression analysis10.1109/34.908962 ↗
22. Scherer, K.R., and Ekman, P. (1982). Handbook of Methods in Nonverbal Behavior Research, Cambridge University Press.
23. Yin, L., Wei, X., Longo, P., and Bhuvanesh, A. (2006, January 20–24). Analyzing facial expressions using intensity-variant 3D data for human computer interaction. Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, China.
24. Soyel, H., and Demirel, H. (2007, January 22–24). Facial expression recognition using 3D facial feature distances. Proceedings of the International Conference on Image Analysis and Recognition, Montreal, QC, Canada.10.5772/6189 ↗
25. Sun, Y., and Yin, L. (2008). Facial expression recognition based on 3D dynamic range model sequences. European Conference on Computer Vision, Springer.10.1007/978-3-540-88688-4_5 ↗
26. Bilinear models for 3-D face and facial expression recognition10.1109/TIFS.2008.924598 ↗
27. Soyel, H., and Demirel, H. (2009, January 2–4). Optimal feature selection for 3D facial expression recognition with geometrically localized facial features. Proceedings of the 2009 Fifth International Conference on Soft Computing, Computing with Words and Perceptions in System Analysis, Decision and Control, Famagusta, Cyprus.10.1109/ICSCCW.2009.5379476 ↗
28. Gong, B., and Wang, Y. (2009, January 19–24). Automatic facial expression recognition on a single 3D face by exploring shape deformation. Proceedings of the 17th ACM International Conference on Multimedia, Beijing, China.10.1145/1631272.1631358 ↗
29. A novel approach to classification of facial expressions from 3D-mesh datasets using modified PCA10.1016/j.patrec.2009.04.007 ↗
30. Zhao, X., Huang, D., Dellandrea, E., and Chen, L. (2010, January 23–26). Automatic 3D facial expression recognition based on a bayesian belief net and a statistical facial feature model. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.10.1109/ICPR.2010.907 ↗
31. Sun, Y., Reale, M., and Yin, L. (2008, January 17–19). Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. Proceedings of the 2008 8th IEEE International Conference on Automatic Face Gesture Recognition, Amsterdam, The Netherlands.10.1109/AFGR.2008.4813336 ↗
32. Zhao, X., Dellandréa, E., Chen, L., and Samaras, D. (2010, January 27–29). AU recognition on 3D faces based on an extended statistical facial feature model. Proceedings of the 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS), Washington, DC, USA.10.1109/BTAS.2010.5634484 ↗
33. Savran, A., Sankur, B., and Bilge, M.T. (2010, January 13–18). Facial action unit detection: 3D versus 2D modality. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition—Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543263 ↗
34. Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time10.1016/j.cub.2013.11.064 ↗
35. Plutchik, R. (1984). Emotions: A general psychoevolutionary theory. Approaches to Emotion, Psychology Press.
36. The Nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice10.1511/2001.28.344 ↗
37. A circumplex model of affect10.1037/h0077714 ↗
38. Relativity in the perception of emotion in facial expressions10.1037/0096-3445.116.3.223 ↗
39. Parrott, W.G. (2001). Emotions in Social Psychology: Essential Readings, Psychology Press.
40. Fabiano, D., and Canavan, S. (2018, January 7–10). Spontaneous and Non-spontaneous 3D facial expression recognition using a statistical model with global and local constraints. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451171 ↗
41. Zhang, X., Yin, L., Cohn, J., Canavan, S., Reale, M., Horowitz, A., and Liu, P. (2013, January 22–26). A High-resolution spontaneous 3D dynamic facial expression database. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.10.1109/FG.2013.6553788 ↗
42. BP4D-Spontaneous: A high-resolution spontaneous 3D dynamic facial expression database10.1016/j.imavis.2014.06.002 ↗
43. 3D facial expression recognition based on Kinect
44. Le, V., Tang, H., and Huang, T.S. (2011, January 21–25). Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. Proceedings of the Face and Gesture 2011, Santa Barbara, CA, USA.10.1109/FG.2011.5771435 ↗
45. Sandbach, G., Zafeiriou, S., Pantic, M., and Rueckert, D. (2011, January 21–25). A dynamic approach to the recognition of 3D facial expressions and their temporal models. Proceedings of the Face and Gesture 2011, Santa Barbara, CA, USA.10.1109/FG.2011.5771434 ↗
46. Recognition of 3D facial expression dynamics10.1016/j.imavis.2012.01.006 ↗
47. Utilizing 3D flow of points for facial expression recognition10.1007/s11042-012-1322-7 ↗
48. Towards a robust affect recognition: Automatic facial expression recognition in 3D faces10.1016/j.eswa.2014.10.042 ↗
49. Jan, A. (2015, January 4–8). Hongying Meng Automatic 3D facial expression recognition using geometric and textured feature fusion. Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia.10.1109/FG.2015.7284860 ↗
50. Hussain, N., Ujir, H., Hipiny, I., and Minoi, J.-L. (2017). 3D Facial action units recognition for emotional expression. arXiv, Available online: https://arxiv.org/abs/1712.00195.
51. Jan, A., Ding, H., Meng, H., Chen, L., and Li, H. (2018, January 15–19). Accurate facial parts localization and deep learning for 3D facial expression recognition. Proceedings of the 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00075 ↗
52. Drira, H., Amor, B.B., Daoudi, M., Srivastava, A., and Berretti, S. (2012, January 11–15). 3D dynamic expression recognition based on a novel Deformation Vector Field and Random Forest. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan.
53. Huynh, X.-P., Tran, T.-D., and Kim, Y.-G. (2016). Convolutional neural network models for facial expression recognition using BU-3DFE database. Information Science and Applications (ICISA), Springer.10.1007/978-981-10-0557-2_44 ↗
54. An expression recognition algorithm based on convolution neural network and RGB-D Images10.1051/matecconf/201817303066 ↗
55. Zeng, W., Li, H., Chen, L., Morvan, J., and Gu, X.D. (2013, January 22–26). An automatic 3D expression recognition framework based on sparse representation of conformal images. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.10.1109/FG.2013.6553749 ↗
56. An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition10.1016/j.cviu.2015.07.005 ↗
57. Tang, H., and Huang, T.S. (2008, January 23–28). 3D facial expression recognition based on automatically selected features. Proceedings of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2008, Anchorage, AK, USA.
58. Tang, H., Fu, Y., Tu, J., Huang, T.S., and Hasegawa-Johnson, M. (2008, January 7–9). EAVA: A 3D emotive audio-visual avatar. Proceedings of the 2008 IEEE Workshop on Applications of Computer Vision, Copper Mountain, CO, USA.10.1109/WACV.2008.4544003 ↗
59. Soyel, H., and Demirel, H. (2008, January 27–29). 3D facial expression recognition with geometrically localized facial features. Proceedings of the 2008 23rd International Symposium on Computer and Information Sciences, Istanbul, Turkey.10.1109/ISCIS.2008.4717898 ↗
60. Maalej, A., Amor, B.B., Daoudi, M., Srivastava, A., and Berretti, S. (2010, January 23–26). Local 3D shape analysis for facial expression recognition. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.10.1109/ICPR.2010.1003 ↗
61. Li, H., Morvan, J.-M., and Chen, L. (2011, January 22–25). 3D facial expression recognition based on histograms of surface differential quantities. International Conference on Advanced Concepts for Intelligent Vision Systems, Ghent, Belgium.10.1007/978-3-642-23687-7_44 ↗
62. Vezzetti, E., Tornincasa, S., Moos, S., Marcolin, F., Violante, M.G., Speranza, D., Buisan, D., and Padula, F. (2016). 3D Human Face Analysis: Automatic expression recognition. Biomed. Eng.10.2316/P.2016.832-067 ↗
63. 3D face expression recognition via geometry A preparatory path
64. Chen, H., Li, J., Zhang, F., Li, Y., and Wang, H. (2015, January 8–10). 3D model-based continuous emotion recognition. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
65. Zhen, Q., Huang, D., Wang, Y., and Chen, L. (2015). Muscular movement model based automatic 3D facial expression recognition. International Conference on MultiMedia Modeling, Springer.10.1007/978-3-319-14445-0_45 ↗
66. Muscular movement model-based automatic 3D/4D facial expression recognition10.1109/TMM.2016.2557063 ↗
67. Berretti, S., Bimbo, A.D., Pala, P., Amor, B.B., and Daoudi, M. (2010, January 23–26). A Set of Selected SIFT Features for 3D Facial Expression Recognition. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.10.1109/ICPR.2010.1002 ↗
68. Real-time 2D+3D facial action and expression recognition10.1016/j.patcog.2009.12.009 ↗
69. Venkatesh, Y.V., Kassim, A.K., and Murthy, O.V.R. (2010, January 23–26). Resampling Approach to Facial Expression Recognition Using 3D Meshes. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.10.1109/ICPR.2010.919 ↗
70. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints10.1007/s00371-011-0611-x ↗
71. Vretos, N., Nikolaidis, N., and Pitas, I. (2011, January 11–14). 3D facial expression recognition using Zernike moments on depth images. Proceedings of the 2011 18th IEEE International Conference on Image Processing, Brussels, Belgium.10.1109/ICIP.2011.6116669 ↗
72. Li, H., Chen, L., Huang, D., Wang, Y., and Morvan, J. (2012, January 11–15). 3D facial expression recognition via multiple kernel learning of Multi-Scale Local Normal Patterns. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan.
73. 3D facial expression recognition using maximum relevance minimum redundancy geometrical features10.1186/1687-6180-2012-213 ↗
74. Rajamanoharan, G., Zafeiriou, S., and Pantic, M. (2012, January 3–7). Binary pattern analysis for 3D facial action unit detection. Proceedings of the British Machine Vision Conference (BMVC), Guildford, UK.
75. Lemaire, P., Ardabilian, M., Chen, L., and Daoudi, M. (2013, January 22–26). Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.10.1109/FG.2013.6553821 ↗
76. An automatic framework for textured 3D video-based facial expression recognition10.1109/TAFFC.2014.2330580 ↗
77. Expression-robust 3D face recognition via weighted sparse representation of multi-scale and multi-component local normal patterns10.1016/j.neucom.2013.11.018 ↗
78. Xue, M., Mian, A., Liu, W., and Li, L. (2014, January 24–26). Fully automatic 3D facial expression recognition using local depth features. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Steamboat Springs, CO, USA.
79. Feature selection for improved 3D facial expression recognition10.1016/j.patrec.2013.10.026 ↗
80. Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns10.1016/j.sigpro.2014.09.033 ↗
81. Using Kinect for real-time emotion recognition via facial expressions10.1631/FITEE.1400209 ↗
82. Yang, X., Huang, D., Wang, Y., and Chen, L. (2015, January 4–8). Automatic 3D facial expression recognition using geometric scattering representation. Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia.
83. Li, X., Ruan, Q., and An, G. (2013, January 22–25). Analysis of range images used in 3D facial expression recognition. Proceedings of the 2013 IEEE International Conference of IEEE Region 10 (TENCON 2013), Xi’an, China.10.1109/TENCON.2013.6718813 ↗
84. Derkach, D., and Sukno, F.M. (June, January 30). Local shape spectrum analysis for 3D facial expression recognition. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.10.1109/FG.2017.143 ↗
85. Non-rigid registration based model-free 3D facial expression recognition10.1016/j.cviu.2017.07.005 ↗
86. 3D facial expression recognition using kernel methods on Riemannian manifold10.1016/j.engappai.2017.05.009 ↗
87. Dong, Z., Jia, X., Gao, W., and Wang, K. (2017, January 16–17). Training on statistical feature models of action units for 3D facial expression recognition. Proceedings of the Information Science and Cloud Computing (ISCC 2017), Guangzhou, China.10.22323/1.300.0021 ↗
88. Ramanathan, S., Kassim, A., Venkatesh, Y.V., and Wah, W.S. (2006, January 8–11). Human Facial Expression Recognition using a 3D Morphable Model. Proceedings of the 2006 International Conference on Image Processing, Atlanta, GA, USA.10.1109/ICIP.2006.312417 ↗
89. 3D/4D facial expression analysis: An advanced annotated face model approach10.1016/j.imavis.2012.02.004 ↗
90. Deep learning10.1038/nature14539 ↗
91. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
92. (2019, July 31). Deep Learning: 3 Cose da Sapere. Available online: https://it.mathworks.com/discovery/deep-learning.html.
93. Walecki, R., Rudovic, O., Pavlovic, V., Schuller, B., and Pantic, M. (2017, January 21-26). Deep structured learning for facial action unit intensity estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.605 ↗
94. (2019, July 31). Rete Neurale Convoluzionale. Available online: https://it.mathworks.com/solutions/deep-learning/convolutional-neural-network.html.
95. Backpropagation applied to handwritten zip code recognition10.1162/neco.1989.1.4.541 ↗
96. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
97. Darwin, C., and Prodger, P. (1998). The Expression of the Emotions in Man and Animals, Oxford University Press.10.1093/oso/9780195112719.002.0002 ↗
98. Bartlett, M.S., Littlewort, G., Fasel, I., and Movellan, J.R. (2003, January 16–22). Real time face detection and facial expression recognition: Development and applications to human computer interaction. Proceedings of the 2003 Conference on Computer Vision and Pattern Recognition Workshop, Madison, WI, USA.10.1109/CVPRW.2003.10057 ↗
99. Robust real-time object detection
100. Automatic facial expression recognition using DCNN10.1016/j.procs.2016.07.233 ↗