Comparative Analysis of Emotion Classification Based on Facial Expression and Physiological Signals Using Deep Learning
Source abstract
Abstract
This study aimed to classify emotion based on facial expression and physiological signals using deep learning and to compare the analyzed results. We asked 53 subjects to make facial expressions, expressing four types of emotion. Next, the emotion-inducing video was watched for 1 min, and the physiological signals were obtained. We defined four emotions as positive and negative emotions and designed three types of deep-learning models that can classify emotions. Each model used facial expressions and physiological signals as inputs, and a model in which these two types of input were applied simultaneously was also constructed. The accuracy of the model was 81.54% when physiological signals were used, 99.9% when facial expressions were used, and 86.2% when both were used. Constructing a deep-learning model with only facial expressions showed good performance. The results of this study confirm that the best approach for classifying emotion is using only facial expressions rather than data from multiple inputs. However, this is an opinion presented only in terms of accuracy without considering the computational cost, and it is suggested that physiological signals and multiple inputs be used according to the situation and research purpose.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · EEG and Brain-Computer Interfaces · Infant Health and Development
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Kinesiology, University of Maryland, College Park, MD 20742, USA
Neuromechanics Research Core, University of Maryland, College Park, MD 20742, USA
Department of Human-Centered Artificial Intelligence, Sangmyung University, Seoul 03016, Korea
Institute of Intelligence Informatics Technology, Sangmyung University, Seoul 03016, Korea
Funding
National Research Foundation of Korea
References (48)
Show source-provided references
1. Emotion Recognition Involving Physiological and Speech Signals: A Com-prehensive Review
2. Kordic, V. (2010). Emotion Recognition through Physiological Signals for Human-Machine Communication. Cutting Edge Robotics 2010, IntechOpen.10.5772/288 ↗
3. Oh, S., Lee, J.-Y., and Kim, D.K. (2020). The Design of CNN Architectures for Optimal Six Basic Emotion Classification Using Multiple Physiological Signals. Sensors, 20.10.3390/s20030866 ↗
4. Wagner, H., and Manstead, A. (1989). The argument and evidence about universals in facial expressions of emotion. Handbooks of Psychophysiology, John Wiley & Sons.
5. Ponnambalam, S.G., Parkkinen, J., and Ramanathan, K.C. (2012). Supplier Selection: Reliability Based Total Cost of Ownership Approach Using Cuckoo Search. Trends in Intelligent Robotics, Automation, and Manufacturing, Proceedings of the First International Conference, IRAM 2012, Kuala Lumpur, Malaysia, 28–30 November 2012, Springer.
6. Tomkins, S.S. (2008). The Positive Affects. Affect Imagery Consciousness: The Complete Edition, Springer.
7. Tomkins, S.S. (2008). Anger and Fear. Affect Imagery Consciousness: The Complete Edition, Springer.
8. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice10.1511/2001.28.344 ↗
9. (2021, May 09). The American Psychological Association, Dimensional theory of emotion in APA Dictionary of Psychology. Available online: https://dictionary.apa.org/dimensional-theory-of-emotion.
10. A Circumplex Model of Affect10.1037/h0077714 ↗
11. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament10.1007/BF02686918 ↗
12. Objective measures, sensors and computational techniques for stress recognition: A survey10.1016/j.cmpb.2012.07.003 ↗
13. The cornell medical index as an epidemiological tool10.2105/AJPH.56.2.287 ↗
14. Stress status classification based on EEG signals10.7236/JIIBC.2016.16.3.103 ↗
15. Oh, S., and Kim, D.K. (2019, January 1–2). Development of the Stacked ensemble model for convenient emotion classification using respiratory signals. Proceedings of the 2019 International Conference and Exhibition on Computational Biology and Bioinformatics (ICECBB 2019), Taipei, Taiwan.10.1109/ICBCB.2019.8854650 ↗
16. Jerritta, S., Murugappan, M., Nagarajan, R., and Wan, K. (2011, January 4–6). Physiological signals based human emotion recognition: A review. Proceedings of the 2011 IEEE 7th International Colloquium on Signal Processing and its Applications, Penang, Malaysia.10.1109/CSPA.2011.5759912 ↗
17. Development of Squat Posture Guidance System Using Kinect and Wii Balance Board
18. Oh, S., and Kim, D.K. (2019, January 22–24). Posture Classification Model Based on Machine Learning for Guiding of Squat Exercises. Proceedings of the 2019 IERI International Conference on Medical Physics, Medical Engineering and Informatics (ICMMI 2019), Tokyo, Japan.
19. Assistant diagnosis with Chinese electronic medical records based on CNN and BiLSTM with phrase-level and word-level attentions10.1186/s12859-020-03554-x ↗
20. Bharatharaj, J., Huang, L., Al-Jumaily, A.M., Krageloh, C., and Elara, M.R. (2016, January 13–15). Experimental evaluation of parrot-inspired robot and adapted model-rival method for teaching children with autism. Proceedings of the 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV), Phuket, Thailand.10.1109/ICARCV.2016.7838636 ↗
21. Deep learning and time series-to-image encoding for financial forecasting10.1109/JAS.2020.1003132 ↗
22. Narvekar, C., and Rao, M. (2020, January 3–5). Flower classification using CNN and transfer learning in CNN-Agriculture Perspective. Proceedings of the 2020 3rd International Conference on Intelligent Sustainable Systems (ICISS), Thoothukudi, India.10.1109/ICISS49785.2020.9316030 ↗
23. Autonomic nervous system activity in emotion: A review10.1016/j.biopsycho.2010.03.010 ↗
24. Emotion recognition from physiological signals using wireless sensors for presence technologies10.1007/s10111-003-0143-x ↗
25. Voluntary facial action generates emotion-specific autonomic nervous system activity10.1111/j.1469-8986.1990.tb02330.x ↗
26. A meta-analysis of heart rate variability and neuroimaging studies10.1016/j.neubiorev.2011.11.009 ↗
27. Impaired cardiovascular recovery following stress predicts 3-year increases in blood pressure10.1097/01.hjh.0000160208.66405.a8 ↗
28. Automatic stress classification with pupil diameter analysis10.1080/10447318.2013.848320 ↗
29. Development of a Stress Classification Model Using Deep Belief Networks for Stress Monitoring10.4258/hir.2017.23.4.285 ↗
30. Li, R., and Liu, Z. (2020). Stress detection using deep neural networks. BMC Med. Inform. Decis. Mak., 20.10.1186/s12911-020-01299-4 ↗
31. Bobade, P., and Vani, M. (2020, January 15–17). Stress Detection with Machine Learning and Deep Learning using Multimodal Physiological Data. Proceedings of the 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India.10.1109/ICIRCA48905.2020.9183244 ↗
32. Wijsman, J., Grundlehner, B., Liu, H., Hermens, H., and Penders, J. (September, January 30). Towards Mental Stress Detection using wearable physiological sensors. Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA.10.1109/IEMBS.2011.6090512 ↗
33. Sandulescu, V., Andrews, S., Ellis, D., Bellotto, N., and Mozos, O.M. (2015). Stress Detection Using Wearable Physiological Sensors. Artificial Computation in Biology and Medicine, Proceedings of the International Work-Conference on the Interplay Between Natural and Artificial Computation (IWINAC) 2015, Elche, Spain, 1–5 June 2015, Springer.10.1007/978-3-319-18914-7_55 ↗
34. A Convolution Bidirectional Long Short-Term Memory Neural Network for Driver Emotion Recognition10.1109/TITS.2020.3007357 ↗
35. Human stress classification using EEG signals in response to music tracks10.1016/j.compbiomed.2019.02.015 ↗
36. Classification of mental stress levels by analyzing fNIRS signal using linear and non-linear features10.15171/icnj.2018.11 ↗
37. Blanco, J.A., Vanleer, A.C., Calibo, T.K., and Firebaugh, S.L. (2019). Single-Trial Cognitive Stress Classification Using Portable Wireless Electroencephalography. Sensors, 19.10.3390/s19030499 ↗
38. Gao, H., Yüce, A., and Thiran, J. (2014, January 27–30). Detecting emotional stress from facial expressions for driving safety. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP 2014), Paris, France.10.1109/ICIP.2014.7026203 ↗
39. Zhang, H., Feng, L., Li, N., Jin, Z., and Cao, L. (2020). Video-Based Stress Detection through Deep Learning. Sensors, 20.10.3390/s20195552 ↗
40. Facial Emotion Detection Using Neural Network
41. Gavrilescu, M., and Vizireanu, N. (2019). Predicting Depression, Anxiety, and Stress Levels from Videos Using the Facial Action Coding System. Sensors, 19.10.3390/s19173693 ↗
42. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., and Narayanan, S. (2004, January 13–15). Analysis of emotion recognition using facial expressions, speech and multimodal information. Proceedings of the ICMI ‘04: 6th international conference on Multimodal interfaces, State College, PA, USA.10.1145/1027933.1027968 ↗
43. Emotion recognition using deep learning approach from audio–visual emotional big data10.1016/j.inffus.2018.09.008 ↗
44. Bayoudh, K., Knani, R., Hamdaoui, F., and Mtibaa, A. (2021). A survey on deep multimodal learning for computer vision: Advances, trends, applications, and datasets. Vis. Comput., 1–32.10.1007/s00371-021-02166-7 ↗
45. Romaissa, B.D., Mourad, O., and Brahim, N. (2021, January 10–15). Vision-Based Multi-Modal Framework for Action Recognition. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9412863 ↗
46. (2021, January 22). Face Recognition, GitHub. Available online: https://github.com/ageitgey/face_recognition.
47. Machine–Deep–Ensemble Learning Model for Classifying Cybersickness Caused by Virtual Reality Immersion10.1089/cyber.2020.0613 ↗
48. Discrimination between spontaneous and posed smile: Humans versus computers