The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition
Source abstract
Abstract
The convolutional neural network (CNN) has been widely used in image recognition field due to its good performance. This paper proposes a facial expression recognition method based on the CNN model. Regarding the complexity of the hierarchic structure of the CNN model, the activation function is its core, because the nonlinear ability of the activation function really makes the deep neural network have authentic artificial intelligence. Among common activation functions, the ReLu function is one of the best of them, but it also has some shortcomings. Since the derivative of the ReLu function is always zero when the input value is negative, it is likely to appear as the phenomenon of neuronal necrosis. In order to solve the above problem, the influence of the activation function in the CNN model is studied in this paper. According to the design principle of the activation function in CNN model, a new piecewise activation function is proposed. Five common activation functions (i.e., sigmoid, tanh, ReLu, leaky ReLus and softplus–ReLu, plus the new activation function) have been analysed and compared in facial expression recognition tasks based on the Keras framework. The Experimental results on two public facial expression databases (i.e., JAFFE and FER2013) show that the convolutional neural network based on the improved activation function has a better performance than most-of-the-art activation functions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face and Expression Recognition · Machine Learning and ELM · Advanced Computing and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Control Science and Engineering, Shandong University, Jinan 250061, China
School of Mechanical, Electrical and Information Engineering, Shandong University, Weihai 264209, China
Funding
undefined undefined · Yong Song
References (48)
Show source-provided references
1. Mehrabian, A., and Russell, J.A. (1974). An Approach to Environmental Psychology, The MIT Press.
2. Region attention networks for pose and occlusion robust facial expression recognition10.1109/TIP.2019.2956143 ↗
3. Fashioning the face: Sensorimotor simulation contributes to facial expression recognition10.1016/j.tics.2015.12.010 ↗
4. Otwell, K. (2019). Facial Expression Recognition in Educational Learning Systems. (10,319,249), U.S. Patent.
5. Affective computing and sentiment analysis10.1109/MIS.2016.31 ↗
6. Fusing audio, visual and textual clues for sentiment analysis from multimodal content10.1016/j.neucom.2015.01.095 ↗
7. Chaturvedi, I., Satapathy, R., Cavallari, S., and Cambria, E. (2019). Fuzzy commonsense reasoning for multimodal sentiment analysis. Pattern Recognit. Lett., 125.10.1016/j.patrec.2019.04.024 ↗
8. Charniak, E. (2019). Introduction to Deep Learning, The MIT Press.
9. Howard, A.G. (2013). Some improvements on deep convolutional neural network based image classification. arXiv.
10. Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique10.1109/TMI.2016.2553401 ↗
11. Mahdianpari, M., Salehi, B., Rezaee, M., Mohammadimanesh, F., and Zhang, Y. (2018). Very deep convolutional neural networks for complex land cover mapping using multispectral remote sensing imagery. Remote Sens., 10.10.3390/rs10071119 ↗
12. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., and Baskurt, A. (2011). Sequential deep learning for human action recognition. International Workshop on Human Behavior Understanding, Springer.10.1007/978-3-642-25446-8_4 ↗
13. Convolutional neural network for simultaneous prediction of several soil properties using visible/near-infrared, mid-infrared, and their combined spectra10.1016/j.geoderma.2019.06.016 ↗
14. Savvides, M., Luu, K., Zheng, Y., and Zhu, C. (2019). Methods and Software for Detecting Objects in Images Using a Multiscale Fast Region-Based Convolutional Neural Network. (10,354,362), U.S. Patent.
15. Research on a learning rate with energy index in deep learning10.1016/j.neunet.2018.12.009 ↗
16. Luo, L., Xiong, Y., Liu, Y., and Sun, X. (2019). Adaptive gradient methods with dynamic bound of learning rate. arXiv.
17. Yedida, R., and Saha, S. (2019). A novel adaptive learning rate scheduler for deep neural networks. arXiv.
18. Cai, S., Gao, J., Zhang, M., Wang, W., Chen, G., and Ooi, B.C. (2019). Effective and efficient dropout for deep convolutional neural networks. arXiv.
19. Labach, A., Salehinejad, H., and Valaee, S. (2019). Survey of Dropout Methods for Deep Neural Networks. arXiv.
20. Hou, S., and Wang, Z. (February, January 27). Weighted channel dropout for regularization of deep convolutional neural network. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
21. Maguolo, G., Nanni, L., and Ghidoni, S. (2019). Ensemble of Convolutional Neural Networks Trained with Different Activation Functions. arXiv.
22. Dubey, A.K., and Jain, V. (2019). Comparative Study of Convolution Neural Network’s ReLu and Leaky-ReLu Activation Functions. Applications of Computing, Automation and Wireless Systems in Electrical Engineering, Springer.10.1007/978-981-13-6772-4_76 ↗
23. Linearized sigmoidal activation: A novel activation function with tractable non-linear characteristics to boost representation capability10.1016/j.eswa.2018.11.042 ↗
24. Lohani, H.K., Dhanalakshmi, S., and Hemalatha, V. (2019). Performance Analysis of Extreme Learning Machine Variants with Varying Intermediate Nodes and Different Activation Functions. Cognitive Informatics and Soft Computing, Springer.10.1007/978-981-13-0617-4_59 ↗
25. A comparison of deep networks with ReLu activation function and linear spline-type methods10.1016/j.neunet.2018.11.005 ↗
26. Deep learning for source camera identification on mobile devices10.1016/j.patrec.2018.01.005 ↗
27. A modified leaky ReLu scheme (MLRS) for topology optimization with multiple materials
28. Zuo, Z., Li, J., Wei, B., Yang, L., Fei, C., and Naik, N. (2019, January 23–26). Adaptive Activation Function Generation Through Fuzzy Inference for Grooming Text Categorisation. Proceedings of the 2019 IEEE International Conference on Fuzzy Systems, New Orleans, LA, USA.
29. Tsai, Y.H., Jheng, Y.J., and Tsaih, R.H. (2019, January 14–19). The Cramming, Softening and Integrating Learning Algorithm with Parametric ReLu Activation Function for Binary Input/Output Problems. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.10.1109/IJCNN.2019.8852023 ↗
30. Prediction of vehicle casualties in major traffic accidents based on neural network10.1063/1.5090752 ↗
31. Performance surfaces of a single-layer perceptron10.1109/72.80252 ↗
32. Extreme learning machine for multilayer perceptron10.1109/TNNLS.2015.2424995 ↗
33. Akçay, S., Kundegorski, M.E., Devereux, M., and Breckon, T.P. (2016, January 25–28). Transfer learning using convolutional neural networks for object classification within x-ray baggage security imagery. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7532519 ↗
34. Lyons, M.J., Akamatsu, S., Kamachi, M., Gyoba, J., and Budynek, J. (1998, January 14–16). The Japanese female facial expression (JAFFE) database. Proceedings of the Third International Conference on Automatic Face and Gesture Recognition, Nara, Japan.
35. Carrier, P.L., Courville, A., Goodfellow, I.J., Mirza, M., and Bengio, Y. (2013). FER-2013 Face Database, Technical report; Universit de Montral.
36. Facial Expression Identification System Using fisher linear discriminant analysis and K-Nearest Neighbor Methods
37. Alif, M.M.F., Syafeeza, A., Marzuki, P., and Alisa, A.N. (2018, January 8). Fused convolutional neural network for facial expression recognition. Proceedings of the Symposium on Electrical, Mechatronics and Applied Science 2018, Melaka, Malaysia.
38. Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order10.1016/j.patcog.2016.07.026 ↗
39. Liu, P., Han, S., Meng, Z., and Tong, Y. (2014, January 24–27). Facial expression recognition via a boosted deep belief network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, America.10.1109/CVPR.2014.233 ↗
40. Facial expression recognition based on local binary patterns: A comprehensive study10.1016/j.imavis.2008.08.005 ↗
41. Selitskaya, N., Sielicki, S., Jakaite, L., Schetinin, V., Evans, F., Conrad, M., and Sant, P. (2020). Deep Learning for Biometric Face Recognition: Experimental Study on Benchmark Data Sets. Deep Biometrics, Springer.10.1007/978-3-030-32583-1_5 ↗
42. Yee, S.Y., Rassem, T.H., Mohammed, M.F., and Awang, S. (2020). Face Recognition Using Laplacian Completed Local Ternary Pattern (LapCLTP). Advances in Electronics Engineering, Springer.10.1007/978-981-15-1289-6_29 ↗
43. Nandi, A., Dutta, P., and Nasir, M. (2020). Recognizing Human Emotions from Facial Images by Landmark Triangulation: A Combined Circumcenter-Incenter-Centroid Trio Feature-Based Method. Algorithms in Machine Learning Paradigms, Springer.10.1007/978-981-15-1041-0_9 ↗
44. Shin, M., Kim, M., and Kwon, D.S. (2016, January 26–31). Baseline CNN structure analysis for facial expression recognition. Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA.10.1109/ROMAN.2016.7745199 ↗
45. Alfakih, A., Yang, S., and Hu, T. (2019). Multi-view Cooperative Deep Convolutional Network for Facial Recognition with Small Samples Learning. International Symposium on Distributed Computing and Artificial Intelligence, Springer.10.1007/978-3-030-23887-2_24 ↗
46. Yar, H., Jan, T., Hussain, A., and Din, S.U. (2020, March 09). Real-Time Facial Emotion Recognition and Gender Classification for Human Robot Interaction Using CNN. Available online: https://www.academia.edu/41996316/Real-Time_Facial_Emotion_Recognition_and_Gender_Classification_for_Human_Robot_Interaction_using_CNN.
47. Using CNN for facial expression recognition: A study of the effects of kernel size and number of filters on accuracy10.1007/s00371-019-01630-9 ↗
48. Vishwakarma, V.P., and Dalal, S. (2020). A novel non-linear modifier for adaptive illumination normalization for robust face recognition. Multimed. Tools Appl., 1–27.10.1007/s11042-019-08537-6 ↗