Facial Expression Recognition Using Computer Vision: A Systematic Review
Source abstract
Abstract
Emotion recognition has attracted major attention in numerous fields because of its relevant applications in the contemporary world: marketing, psychology, surveillance, and entertainment are some examples. It is possible to recognize an emotion through several ways; however, this paper focuses on facial expressions, presenting a systematic review on the matter. In addition, 112 papers published in ACM, IEEE, BASE and Springer between January 2006 and April 2019 regarding this topic were extensively reviewed. Their most used methods and algorithms will be firstly introduced and summarized for a better understanding, such as face detection, smoothing, Principal Component Analysis (PCA), Local Binary Patterns (LBP), Optical Flow (OF), Gabor filters, among others. This review identified a clear difficulty in translating the high facial expression recognition (FER) accuracy in controlled environments to uncontrolled and pose-variant environments. The future efforts in the FER field should be put into multimodal systems that are robust enough to face the adversities of real world scenarios. A thorough analysis on the research done on FER in Computer Vision based on the selected papers is presented. This review aims to not only become a reference for future research on emotion recognition, but also to provide an overview of the work done in this topic for potential readers.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face and Expression Recognition · Emotion and Mood Recognition · Face recognition and analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
IEETA/DETI, University of Aveiro, 3810-193 Aveiro, Portugal
References (100+)
Show source-provided references
1. Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer.
2. Deep learning10.1038/nature14539 ↗
3. Frontal EEG asymmetry as a moderator and mediator of emotion10.1016/j.biopsycho.2004.03.002 ↗
4. A survey on face detection in the wild: Past, present and future10.1016/j.cviu.2015.03.015 ↗
5. Perez, L., and Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv.
6. Recognizing action units for facial expression analysis10.1109/34.908962 ↗
7. Boser, B.E., Guyon, I.M., and Vapnik, V.N. (1992, January 27–29). A training algorithm for optimal margin classifiers. Proceedings of the Fifth Annual Workshop on Computational Learning Theory, Pittsburgh, PA, USA.10.1145/130385.130401 ↗
8. A survey of deep neural network architectures and their applications10.1016/j.neucom.2016.12.038 ↗
9. (2019, September 26). ACM Digital Library. Available online: https://dl.acm.org/.
10. (2019, September 26). IEEE Xplore Digital Library. Available online: https://ieeexplore.ieee.org/Xplore/home.jsp.
11. (2019, September 26). Bielefeld Academic Search Engine. Available online: https://www.base-search.net/.
12. (2019, September 26). Springer Link. Available online: https://link.springer.com/.
13. Valstar, M.F., Pantic, M., Ambadar, Z., and Cohn, J.F. (2006, January 2–4). Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. Proceedings of the 8th International Conference on Multimodal Interfaces, Banff, AB, Canada.10.1145/1180995.1181031 ↗
14. Duthoit, C.J., Sztynda, T., Lal, S.K., Jap, B.T., and Agbinya, J.I. (2008, January 21–23). Optical flow image analysis of facial expressions of human emotion: Forensic applications. Proceedings of the 1st International Conference on Forensic Applications and Techniques in Telecommunications, Information, and Multimedia and Workshop, Adelaide, Australia.10.4108/e-forensics.2008.2743 ↗
15. Simultaneous facial action tracking and expression recognition in the presence of head motion10.1007/s11263-007-0059-7 ↗
16. User and context adaptive neural networks for emotion recognition10.1016/j.neucom.2007.11.043 ↗
17. Sun, X., Rothkrantz, L., Datcu, D., and Wiggers, P. (2009, January 18–19). A Bayesian approach to recognise facial expressions using vector flows. Proceedings of the International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing, Ruse, Bulgaria.10.1145/1731740.1731772 ↗
18. Popa, M., Rothkrantz, L., and Wiggers, P. (2010, January 17–18). Products appreciation by facial expressions analysis. Proceedings of the 11th International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing on International Conference on Computer Systems and Technologies, Sofia, Bulgaria.10.1145/1839379.1839431 ↗
19. Liu, X., Zhang, L., and Yadegar, J. (2011, January 10–13). A multi-modal emotion recognition system for persistent and non-invasive personal health monitoring. Proceedings of the 2nd Conference on Wireless Health, La Jolla.10.1145/2077546.2077577 ↗
20. Facial emotion recognition using context based multimodal approach
21. Cruz, A.C., Bhanu, B., and Thakoor, N. (2012, January 22–26). Facial emotion recognition with expression energy. Proceedings of the 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, USA.10.1145/2388676.2388777 ↗
22. Soladié, C., Salam, H., Pelachaud, C., Stoiber, N., and Séguier, R. (2012, January 22–26). A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection. Proceedings of the 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, USA.10.1145/2388676.2388782 ↗
23. Monkaresi, H., Calvo, R.A., and Hussain, M.S. (2012, January 21–25). Automatic natural expression recognition using head movement and skin color features. Proceedings of the International Working Conference on Advanced Visual Interfaces, Capri Island, Italy.10.1145/2254556.2254678 ↗
24. Biel, J.I., Teijeiro-Mosquera, L., and Gatica-Perez, D. (2012, January 22–26). Facetube: Predicting personality from facial expressions of emotion in online conversational video. Proceedings of the 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, USA.10.1145/2388676.2388689 ↗
25. Nedkov, S., and Dimov, D. (2013, January 28–29). Emotion recognition by face dynamics. Proceedings of the 14th International Conference on Computer Systems and Technologies, Ruse, Bulgaria.10.1145/2516775.2516794 ↗
26. Measuring instant emotions based on facial expressions during computer-based assessment10.1007/s00779-011-0477-y ↗
27. Meng, H., Huang, D., Wang, H., Yang, H., Ai-Shuraifi, M., and Wang, Y. (2013, January 21–25). Depression recognition based on dynamic facial and vocal expression features using partial least square regression. Proceedings of the 3rd ACM International Workshop on Audio/visual Emotion Challenge, Barcelona, Spain.10.1145/2512530.2512532 ↗
28. Gómez Jáuregui, D.A., and Martin, J.C. (2013, January 9–13). Evaluation of vision-based real-time measures for emotions discrimination under uncontrolled conditions. Proceedings of the 2013 on Emotion Recognition in the Wild Challenge and Workshop, Sydney, Australia.10.1145/2531923.2531925 ↗
29. Fuzzy model of dominance emotions in affective computing10.1007/s00521-014-1637-6 ↗
30. Sangineto, E., Zen, G., Ricci, E., and Sebe, N. (2014, January 3–7). We are not all equal: Personalizing models for facial expression analysis with transductive parameter transfer. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654916 ↗
31. Jang, G.J., Jo, A., and Park, J.S. (2014, January 28–31). Video-based emotion identification using face alignment and support vector machines. Proceedings of the Second International Conference on Human-agent Interaction, Tsukuba, Japa.10.1145/2658861.2658943 ↗
32. Zen, G., Sangineto, E., Ricci, E., and Sebe, N. (2014, January 12–16). Unsupervised domain adaptation for personalized facial emotion recognition. Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey.10.1145/2663204.2663247 ↗
33. Rothkrantz, L. (2014, January 27–28). Online emotional facial expression dictionary. Proceedings of the 15th International Conference on Computer Systems and Technologies, Ruse, Bulgaria.10.1145/2659532.2659627 ↗
34. Chao, L., Tao, J., Yang, M., Li, Y., and Wen, Z. (2015, January 26–30). Long short term memory recurrent neural network based multimodal dimensional emotion recognition. Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, Brisbane, Australia.10.1145/2808196.2811634 ↗
35. Emotion recognition during speech using dynamics of multiple regions of the face
36. Nomiya, H., Sakaue, S., and Hochin, T. (2016, January 26–29). Recognition and intensity estimation of facial expression using ensemble classifiers. Proceedings of the 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS), Okayama, Japan.10.1109/ICIS.2016.7550861 ↗
37. Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation10.1109/ACCESS.2016.2628407 ↗
38. Barsoum, E., Zhang, C., Ferrer, C.C., and Zhang, Z. (2016, January 12–16). Training deep networks for facial expression recognition with crowd-sourced label distribution. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan.10.1145/2993148.2993165 ↗
39. A facial expression emotion recognition based human-robot interaction system10.1109/JAS.2017.7510622 ↗
40. Bouzakraoui, M.S., Sadiq, A., and Enneya, N. (2017, January 29–30). A Customer Emotion Recognition through Facial Expression using POEM descriptor and SVM classifier. Proceedings of the 2nd International Conference on Big Data, Cloud and Applications, Tetouan, Morocco.10.1145/3090354.3090436 ↗
41. Elfaramawy, N., Barros, P., Parisi, G.I., and Wermter, S. (2017, January 17–20). Emotion recognition from body expressions with a neural network architecture. Proceedings of the 5th International Conference on Human Agent Interaction, Bielefeld, Germany.10.1145/3125739.3125772 ↗
42. Facial expressions recognition based on cognition and mapped binary patterns10.1109/ACCESS.2018.2816044 ↗
43. SMEConvnet: A convolutional neural network for spotting spontaneous facial micro-expression from long videos10.1109/ACCESS.2018.2879485 ↗
44. Dominant and complementary emotion recognition from still images of faces10.1109/ACCESS.2018.2831927 ↗
45. Slimani, K., Kas, M., El Merabet, Y., Messoussi, R., and Ruichek, Y. (2018, January 27–28). Facial emotion recognition: A comparative analysis using 22 LBP variants. Proceedings of the 2nd Mediterranean Conference on Pattern Recognition and Artificial Intelligence, Rabat, Morocco.10.1145/3177148.3180092 ↗
46. Bernin, A., Müller, L., Ghose, S., Grecos, C., Wang, Q., Jettke, R., von Luck, K., and Vogt, F. (2018, January 26–29). Automatic Classification and Shift Detection of Facial Expressions in Event-Aware Smart Environments. Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, Corfu, Greece.10.1145/3197768.3201527 ↗
47. Real time facial expression recognition using webcam and SDK affectiva10.9781/ijimai.2017.11.002 ↗
48. Facial action units-based image retrieval for facial expression recognition10.1109/ACCESS.2018.2889852 ↗
49. Slimani, K., Lekdioui, K., Messoussi, R., and Touahni, R. (2019, January 28–29). Compound Facial Expression Recognition Based on Highway CNN. Proceedings of the New Challenges in Data Sciences: Acts of the Second Conference of the Moroccan Classification Society, Kenitra, Morocco.10.1145/3314074.3314075 ↗
50. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
51. Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., and Budynek, J. (1998, January 14–16). The Japanese female facial expression (JAFFE) database. Proceedings of the Third International Conference on Automatic Face And Gesture Recognition, Nara, Japan.
52. Yin, L., Wei, X., Sun, Y., Wang, J., and Rosato, M.J. (2006, January 10–12). A 3D facial expression database for facial behavior research. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), Southampton, UK.
53. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.H. (2013). Challenges in representation learning: A report on three machine learning contests. International Conference on Neural Information Processing, Springer.10.1007/978-3-642-42051-1_16 ↗
54. Dhall, A., Ramana Murthy, O., Goecke, R., Joshi, J., and Gedeon, T. (2015, January 9–13). Video and image based emotion recognition challenges in the wild: Emotiw 2015. Proceedings of the 2015 ACM on International Conference On Multimodal Interaction, Seattle, WA, USA.10.1145/2818346.2829994 ↗
55. Pantic, M., Valstar, M., Rademaker, R., and Maat, L. (2005, January 6). Web-based database for facial expression analysis. Proceedings of the 2005 IEEE International Conference on Multimedia and Expo, Amsterdam, Netherlands.
56. Martin, O., Kotsia, I., Macq, B., and Pitas, I. (2006, January 3–7). The eNTERFACE’05 audio-visual emotion database. Proceedings of the 22nd International Conference on Data Engineering Workshops (ICDEW’06), Atlanta, GA, USA.10.1109/ICDEW.2006.145 ↗
57. Facial expressions of emotion (KDEF): Identification under different display-duration conditions10.3758/BRM.40.1.109 ↗
58. Presentation and validation of the Radboud Faces Database10.1080/02699930903485076 ↗
59. (2019, September 08). The Extended Cohn–Kanade Database. Available online: http://www.consortium.ri.cmu.edu/ckagree/.
60. (2019, September 08). The Japanese Female Facial Expression Database. Available online: http://www.kasrl.org/jaffe.html.
61. (2019, September 08). Binghamton University 3D Facial Expression Database. Available online: http://www.cs.binghamton.edu/~lijun/Research/3DFE/3DFE_Analysis.html.
62. (2019, September 08). Facial Expression Recognition 2013 Database. Available online: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data.
63. (2019, September 08). Emotion Recognition in the Wild Database. Available online: https://cs.anu.edu.au/few/AFEW.html.
64. (2019, September 08). MMI Database. Available online: https://mmifacedb.eu/.
65. (2019, September 08). eNTERFACE’05 Audio-Visual Emotion Database. Available online: http://www.enterface.net/enterface05/.
66. (2019, September 08). Karolinska Directed Emotional Faces Database. Available online: http://kdef.se/.
67. (2019, September 08). Radboud Faces Database. Available online: http://www.socsci.ru.nl:8180/RaFD2/RaFD?p=main.
68. Robust real-time face detection10.1023/B:VISI.0000013087.49260.fb ↗
69. Kazemi, V., and Sullivan, J. (2014, January 23–28). One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.241 ↗
70. Joint face detection and alignment using multitask cascaded convolutional networks10.1109/LSP.2016.2603342 ↗
71. Farfade, S.S., Saberian, M.J., and Li, L.J. (2015, January 23–26). Multi-view face detection using deep convolutional neural networks. Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China.10.1145/2671188.2749408 ↗
72. Azulay, A., and Weiss, Y. (2018). Why do deep convolutional networks generalize so poorly to small image transformations?. arXiv.
73. Bilateral filtering for gray and color images
74. On Gabor’s contribution to image enhancement10.1016/0031-3203(94)90013-2 ↗
75. A Comparative Study on Histogram Equalization and Cumulative Histogram Equalization
76. Adaptive histogram equalization and its variations10.1016/S0734-189X(87)80186-X ↗
77. The problem of overfitting10.1021/ci0342472 ↗
78. Jolliffe, I. (2011). Principal Component Analysis, Springer.10.1007/978-3-642-04898-2_455 ↗
79. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns10.1109/TPAMI.2002.1017623 ↗
80. Determining optical flow10.1016/0004-3702(81)90024-2 ↗
81. Barron, J.L., Fleet, D.J., Beauchemin, S.S., and Burkitt, T. (1992, January 15–18). Performance of optical flow techniques. Proceedings of the 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Champaign, IL, USA.
82. Active appearance models10.1109/34.927467 ↗
83. Abdulameer, M.H., Abdullah, S., Huda, S.N., and Othman, Z.A. (2014). A modified active appearance model based on an adaptive artificial bee colony. Sci. World J., 2014.10.1155/2014/879031 ↗
84. Ekman, R. (1997). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press.10.1093/oso/9780195104462.001.0001 ↗
85. Pakstas, A., Forchheimer, R., and Pandzic, I.S. (2002). MPEG-4 Facial Animation: The Standard, Implementation and Applications, John Wiley & Sons, Inc.
86. Real time facial expression recognition system with applications to facial animation in MPEG-4
87. Unsupervised texture segmentation using Gabor filters10.1016/0031-3203(91)90143-S ↗
88. Choraś, R.S. (2010). Image Processing and Communications Challenges 2, Springer.10.1007/978-3-642-16295-4 ↗
89. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
90. A Comparative Study of SIFT and its Variants10.2478/msr-2013-0021 ↗
91. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
92. Liu, Y., Li, Y., Ma, X., and Song, R. (2017). Facial expression recognition with fusion features extracted from salient facial areas. Sensors, 17.10.20944/preprints201701.0102.v1 ↗
93. Dropout: A simple way to prevent neural networks from overfitting
94. Nearest neighbor pattern classification10.1109/TIT.1967.1053964 ↗
95. Bayesian network classifiers10.1023/A:1007465528199 ↗
96. Hidden markov models10.1016/S0959-440X(96)80056-X ↗
97. Induction of decision trees10.1007/BF00116251 ↗
98. Random forests10.1023/A:1010933404324 ↗
99. Chui, K.T., and Lytras, M.D. (2019). A Novel MOGA-SVM Multinomial Classification for Organ Inflammation Detection. Appl. Sci., 9.10.3390/app9112284 ↗
100. A survey of cross-validation procedures for model selection10.1214/09-SS054 ↗