A Unified Framework of Deep Learning-Based Facial Expression Recognition System for Diversified Applications
Source abstract
Abstract
This work proposes a facial expression recognition system for a diversified field of applications. The purpose of the proposed system is to predict the type of expressions in a human face region. The implementation of the proposed method is fragmented into three components. In the first component, from the given input image, a tree-structured part model has been applied that predicts some landmark points on the input image to detect facial regions. The detected face region was normalized to its fixed size and then down-sampled to its varying sizes such that the advantages, due to the effect of multi-resolution images, can be introduced. Then, some convolutional neural network (CNN) architectures were proposed in the second component to analyze the texture patterns in the facial regions. To enhance the proposed CNN model’s performance, some advanced techniques, such data augmentation, progressive image resizing, transfer-learning, and fine-tuning of the parameters, were employed in the third component to extract more distinctive and discriminant features for the proposed facial expression recognition system. The performance of the proposed system, due to different CNN models, is fused to achieve better performance than the existing state-of-the-art methods and for this reason, extensive experimentation has been carried out using the Karolinska-directed emotional faces (KDEF), GENKI-4k, Cohn-Kanade (CK+), and Static Facial Expressions in the Wild (SFEW) benchmark databases. The performance has been compared with some existing methods concerning these databases, which shows that the proposed facial expression recognition system outperforms other competing methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face and Expression Recognition · Emotion and Mood Recognition · Face recognition and analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Aliah University, Kolkata 700156, India
Electrical and Computer Engineering, University of Dayton, 300 College Park, Dayton, OH 45469-0232, USA
Department of Computer Science and Engineering, National Institute of Technology, Srinagar 190006, India
References (74)
Show source-provided references
1. Mollahosseini, A., Chan, D., and Mahoor, M.H. (2016, January 7–10). Going deeper in facial expression recognition using deep neural networks. Proceedings of the 2016 IEEE (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477450 ↗
2. Multimodal human–computer interaction: A survey10.1016/j.cviu.2006.10.019 ↗
3. Kaukomaa, T. (2015). Facial Expressions as an Interactional Resource in Everyday Face-to-Face Conversation, Helsingin Yliopisto.
4. Facial expression recognition using a combination of multiple facial features and support vector machine10.1007/s00500-017-2634-3 ↗
5. Constants across cultures in the face and emotion10.1037/h0030377 ↗
6. Facial expression recognition based on a hybrid model combining deep and shallow features10.1007/s12559-019-09654-y ↗
7. Four not six: Revealing culturally common facial expressions of emotion10.1037/xge0000162 ↗
8. A facial-expression monitoring system for improved healthcare in smart cities10.1109/ACCESS.2017.2712788 ↗
9. Jarwar, M.A., and Chong, I. (2017, January 11–13). Exploiting IoT services by integrating emotion recognition in Web of Objects. Proceedings of the 2017 International Conference on Information Networking (ICOIN), Da Nang, Vietnam.10.1109/ICOIN.2017.7899474 ↗
10. AI Weighs in On Debate about Universal Facial Expressions10.1038/d41586-020-03509-5 ↗
11. Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect10.1075/pc.8.1.09lis ↗
12. Fernández-Dols, J.M., and Russell, J.A. (1997). The Psychology of Facial Expression, Cambridge University Press.
13. Ekman, P., and Rosenberg, E.L. (1997). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press.10.1093/oso/9780195104462.001.0001 ↗
14. Face recognition using fusion of feature learning techniques10.1016/j.measurement.2019.06.008 ↗
15. Umer, S., Rout, R.K., Pero, C., and Nappi, M. (2021). Facial expression recognition with trade-offs between data augmentation and deep learning features. J. Ambient. Intell. Humaniz. Comput., 1–15.10.1007/s12652-020-02845-8 ↗
16. Disfa: A spontaneous facial action intensity database10.1109/T-AFFC.2013.4 ↗
17. Automatic analysis of facial expressions: The state of the art10.1109/34.895976 ↗
18. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of Oriented Gradients for Human Detection. Proceedings of the IEEE Computer Society Conference on (CVPR), San Diego, CA, USA.
19. Sift: Scale invariant feature transform
20. A comparative study of texture measures with classification based on featured distributions10.1016/0031-3203(95)00067-4 ↗
21. Liu, M., Shan, S., Wang, R., and Chen, X. (2014, January 23–28). Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.226 ↗
22. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
23. Emotion recognition from posed and spontaneous dynamic expressions: Human observers versus machine analysis10.1037/emo0000712 ↗
24. Le Cun, Y., Bottou, L., and Bengio, Y. (1997, January 21–24). Reading checks with multilayer graph transformer networks. Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, Munich, Germany.
25. Shojaeilangari, S., Yun, Y.W., and Khwang, T.E. (2011, January 13–16). Person independent facial expression analysis using Gabor features and Genetic Algorithm. Proceedings of the 2011 8th International Conference on Information, Communications & Signal Processing, Singapore.10.1109/ICICS.2011.6173537 ↗
26. Dhall, A., Goecke, R., Lucey, S., and Gedeon, T. (2011, January 6–13). Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130508 ↗
27. Zhao, X., Liang, X., Liu, L., Li, T., Han, Y., Vasconcelos, N., and Yan, S. (2016, January 8–16). Peak-piloted deep network for facial expression recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherland.10.1007/978-3-319-46475-6_27 ↗
28. Recognizing action units for facial expression analysis10.1109/34.908962 ↗
29. Efficient smile detection by extreme learning machine10.1016/j.neucom.2014.04.072 ↗
30. Jain, A.K., and Li, S.Z. (2011). Handbook of Face Recognition, Springer.
31. Eigenfaces for recognition10.1162/jocn.1991.3.1.71 ↗
32. Sonka, M., Hlavac, V., and Boyle, R. (2014). Image Processing, Analysis, and Machine Vision, Cengage Learning.
33. Face recognition using laplacianfaces10.1109/TPAMI.2005.55 ↗
34. Independent comparative study of PCA, ICA, and LDA on the FERET data set10.1002/ima.20059 ↗
35. Yang, M., and Zhang, L. (2010, January 5–11). Gabor feature based sparse representation for face recognition with gabor occlusion dictionary. Proceedings of the European Conference on Computer Vision, Heraklion, Greece.10.1007/978-3-642-15567-3_33 ↗
36. Gutta, S., Wechsler, H., and Phillips, P.J. (1998, January 14–16). Gender and ethnic classification of face images. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan.
37. Zhang, G., and Wang, Y. (2009, January 20–23). Multimodal 2D and 3D facial ethnicity classification. Proceedings of the 2009 Fifth International Conference on Image and Graphics, Xi’an, China.10.1109/ICIG.2009.113 ↗
38. Zhang, Z., Lyons, M., Schuster, M., and Akamatsu, S. (1998, January 14–16). Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan.
39. Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., and Movellan, J. (2005, January 20–25). Recognizing facial expression: Machine learning and application to spontaneous behavior. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
40. Rose, N. (2006, January 10–12). Facial expression classification using gabor and log-gabor filters. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), Southampton, UK.
41. Wu, T., Bartlett, M.S., and Movellan, J.R. (2010, January 13–18). Facial expression recognition using gabor motion energy filters. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543267 ↗
42. Zavarez, M.V., Berriel, R.F., and Oliveira-Santos, T. (2017, January 17–20). Cross-database facial expression recognition based on fine-tuned deep convolutional network. Proceedings of the 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Niteroi, Brazil.10.1109/SIBGRAPI.2017.60 ↗
43. Facial expression recognition using radial encoding of local Gabor features and classifier synthesis10.1016/j.patcog.2011.05.006 ↗
44. Almaev, T.R., and Valstar, M.F. (2013, January 2–5). Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, Switzerland.10.1109/ACII.2013.65 ↗
45. Dynamic texture recognition using local binary patterns with an application to facial expressions10.1109/TPAMI.2007.1110 ↗
46. Zhu, X., and Ramanan, D. (2012, January 16–21). Face detection, pose estimation, and landmark localization in the wild. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
47. Mita, T., Kaneko, T., and Hori, O. (2005, January 17–21). Joint haar-like features for face detection. Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, Beijing, China.10.1109/ICCV.2005.129 ↗
48. Liu, Y., Cao, Y., Li, Y., Liu, M., Song, R., Wang, Y., Xu, Z., and Ma, X. (2016, January 6–10). Facial expression recognition with PCA and LBP features extracting from active facial patches. Proceedings of the 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR), Angkor Wat, Cambodia.10.1109/RCAR.2016.7784056 ↗
49. Tuceryan, M., and Jain, A.K. (1993). Texture Analysis. Handbook of Pattern Recognition & Computer Vision, World Scientific Publishing Company.10.1142/9789814343138_0010 ↗
50. Dumoulin, V., and Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv.
51. Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., and Guibas, L.J. (November, January 27). Kpconv: Flexible and deformable convolution for point clouds. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
52. Motion analysis for image enhancement: Resolution, occlusion, and transparency10.1006/jvci.1993.1030 ↗
53. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. (2016). Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv.
54. Ma, W., and Lu, J. (2017). An equivalence of fully connected layer and convolutional layer. arXiv.
55. Ferro-Pérez, R., and Mitre-Hernandez, H. (2020). ResMoNet: A Residual Mobile-based Network for Facial Emotion Recognition in Resource-Limited Systems. arXiv.
56. Wang, C.Y., Mark Liao, H.Y., Wu, Y.H., Chen, P.Y., Hsieh, J.W., and Yeh, I.H. (2020, January 14–19). CSPNet: A new backbone that can enhance learning capability of cnn. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00203 ↗
57. Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv.
58. Dropout: A simple way to prevent neural networks from overfitting
59. Mescheder, L., Geiger, A., and Nowozin, S. (2018). Which training methods for GANs do actually converge?. arXiv.
60. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv.
61. Kingma, D.P. (2015). Adam: A Method for Stochastic Optimization. arXiv.
62. Scherer, D., Müller, A., and Behnke, S. (2010, January 15–18). Evaluation of pooling operations in convolutional architectures for object recognition. Proceedings of the International Conference on Artificial Neural Networks, Thessaloniki, Greece.10.1007/978-3-642-15825-4_10 ↗
63. Hernández-García, A., and König, P. (2018, January 4–7). Further advantages of data augmentation on convolutional neural networks. Proceedings of the International Conference on Artificial Neural Networks, Rhodes, Greece.10.1007/978-3-030-01418-6_10 ↗
64. Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. (2017, January 21–26). Enhanced deep residual networks for single image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.151 ↗
65. Jain, V., and Crowley, J.L. (2013, January 20–22). Smile detection using multi-scale gaussian derivatives. Proceedings of the 12th WSEAS International Conference on Signal Processing, Robotics and Automation, Cambridge, UK.
66. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
67. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
68. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
69. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
70. Rao, Q., Qu, X., Mao, Q., and Zhan, Y. (2015, January 21–24). Multi-pose facial expression recognition based on SURF boosting. Proceedings of the 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xi’an, China.10.1109/ACII.2015.7344635 ↗
71. Facial expression recognition using lp-norm MKL multiclass-SVM10.1007/s00138-015-0677-y ↗
72. A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios10.1016/j.neucom.2015.10.022 ↗
73. A ROI-guided deep architecture for robust facial expressions recognition10.1016/j.ins.2020.02.047 ↗
74. Liu, M., Li, S., Shan, S., and Chen, X. (2013, January 22–26). Au-aware deep networks for facial expression recognition. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.