journal article·2023-02-16Open access

Review of Studies on Emotion Recognition and Judgment Based on Physiological Signals

Wenqian Lin, Chao Li

Applied Sciences· Volume 13· Issue 4

Abstract

People’s emotions play an important part in our daily life and can not only reflect psychological and physical states, but also play a vital role in people’s communication, cognition and decision-making. Variations in people’s emotions induced by external conditions are accompanied by variations in physiological signals that can be measured and identified. People’s psychological signals are mainly measured with electroencephalograms (EEGs), electrodermal activity (EDA), electrocardiograms (ECGs), electromyography (EMG), pulse waves, etc. EEG signals are a comprehensive embodiment of the operation of numerous neurons in the cerebral cortex and can immediately express brain activity. EDA measures the electrical features of skin through skin conductance response, skin potential, skin conductance level or skin potential response. ECG technology uses an electrocardiograph to record changes in electrical activity in each cardiac cycle of the heart from the body surface. EMG is a technique that uses electronic instruments to evaluate and record the electrical activity of muscles, which is usually referred to as myoelectric activity. EEG, EDA, ECG and EMG have been widely used to recognize and judge people’s emotions in various situations. Different physiological signals have their own characteristics and are suitable for different occasions. Therefore, a review of the research work and application of emotion recognition and judgment based on the four physiological signals mentioned above is offered. The content covers the technologies adopted, the objects of application and the effects achieved. Finally, the application scenarios for different physiological signals are compared, and issues for attention are explored to provide reference and a basis for further investigation.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Emotion and Mood Recognition · EEG and Brain-Computer Interfaces · Color perception and design

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Media and Design, Hangzhou Dianzi University, Hangzhou 310018, China

College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China

Funding

National Natural Science Foundation of China

References (100+)

Show source-provided references
1. Affect detection from multichannel physiology during learning sessions with auto tutor10.1007/978-3-642-21869-9_19 ↗
2. Mood and the speed of decisions about anticipated resources and hazards10.1016/j.evolhumbehav.2010.07.005 ↗
3. Human-computer interaction: Overview on state of the art
4. Multimodal emotion recognition based on manifold learning and convolution neural network10.1007/s11042-022-13149-8 ↗
5. Quispe, K.G.M., Utyiama, D.M.S., dos Santos, E.M., Oliveira, H.A.B.F., and Souto, E.J. (2022). Applying self-supervised representation learning for emotion recognition using physiological signals. Sensors, 22.10.3390/s22239102 ↗
6. Cognitive investigation on the effect of augmented reality-based reading on emotion classification performance: A new dataset10.1016/j.bspc.2022.103942 ↗
7. A framework to model and control the state of presence in virtual reality systems10.1109/TAFFC.2022.3195697 ↗
8. SigRep: Toward robust wearable emotion recognition with contrastive representation learning10.1109/ACCESS.2022.3149509 ↗
9. Emotion recognition with short-period physiological signals using bimodal sparse autoencoders10.32604/iasc.2022.020849 ↗
10. Learning DenseNet features from EEG based spectrograms for subject independent emotion recognition10.1016/j.bspc.2022.103485 ↗
11. Moin, A., Aadil, F., Ali, Z., and Kang, D.W. (2023). Emotion recognition framework using multiple modalities for an effective human-computer interaction. J. Supercomput.10.1007/s11227-022-05026-w ↗
12. WeDea: A new eeg-based framework for emotion recognition10.1109/JBHI.2021.3091187 ↗
13. Multiple instance learning for emotion recognition using physiological signals10.1109/TAFFC.2019.2954118 ↗
14. Modality encoded latent dataset for emotion recognition10.1016/j.bspc.2022.104140 ↗
15. Emotion recognition based on multi-modal physiological signals and transfer learning10.3389/fnins.2022.1000716 ↗
16. Normal inverse gaussian features for EEG-based automatic emotion recognition10.1109/TIM.2022.3205894 ↗
17. Biosignal-based user-independent recognition of emotion and personality with importance weighting10.1007/s11042-022-12711-8 ↗
18. Hasnul, M.A., Ab Aziz, N.A., and Abd Aziz, A. (2023). Augmenting ECG data with multiple filters for a better emotion recognition system. Arab. J. Sci. Eng., 1–22.10.1007/s13369-022-07585-9 ↗
19. A transfer fusion framework for body sensor networks (BSNs): Dynamic domain adaptation from distribution evaluation to domain evaluation10.1016/j.inffus.2022.10.026 ↗
20. EEG-based cross-subject emotion recognition using Fourier-Bessel series expansion based empirical wavelet transform and NCA feature selection method10.1016/j.ins.2022.07.121 ↗
21. Asiain, D., de Leon, J.P., and Beltran, J.R. (2022). MsWH: A multi-sensory hardware platform for capturing and analyzing physiological emotional signals. Sensors, 22.10.3390/s22155775 ↗
22. Zontone, P., Affanni, A., Bernardini, R., Del Linz, L., Piras, A., and Rinaldo, R. (2022). Analysis of physiological signals for stress recognition with different car handling setups. Electronics, 11.10.3390/electronics11060888 ↗
23. The evaluation of automobile interior acceleration sound fused with physiological signal using a hybrid deep neural network10.1016/j.ymssp.2022.109675 ↗
24. Affective image classification basedon usereye movement and EEG experience information10.1093/iwc/iwy018 ↗
25. EEG-based emotion estimation using Bayesian weighted-log-posterior function and perception convergence algorithm10.1016/j.compbiomed.2013.10.017 ↗
26. Andreu-Perez, A.R., Kiani, M., Andreu-Perez, J., Reddy, P., Andreu-Abela, J., Pinto, M., and Izzetoglu, K. (2021). Single-trial recognition of video gamer’s expertise from brainhaemodynamic and facial emotion responses. Brain Sci., 11.10.3390/brainsci11010106 ↗
27. Zhang, J., Zhou, Y.T., and Liu, Y. (2020). EEG-based emotion recognition using an improved radial basis function neural network. J. Ambient Intell. Humaniz. Comput., 5.10.1007/s12652-020-02049-0 ↗
28. Aesthetic preference recognition of 3D shapes using EEG10.1007/s11571-015-9363-z ↗
29. Emotion assessment: Arousal evaluation using EEG’s and peripheral physiological signals10.1007/11848035_70 ↗
30. Performance evaluation of multi-channel electroencephalogram signal (EEG) based time frequency analysis for humane motion recognition10.1016/j.bspc.2022.103966 ↗
31. EEG-based emotion recognition with deep convolutional neural networks10.1515/bmt-2019-0306 ↗
32. DECAF:meg-based multimodal database for decoding affective physiological responses10.1109/TAFFC.2015.2392932 ↗
33. An art therapy evaluation method based on emotion recognition using EEG deep temporal features10.1007/s11042-022-12002-2 ↗
34. Emotionrecognitionusing EEG phase space dynamics and Poincare intersections10.1016/j.bspc.2020.101918 ↗
35. On identification of driving-induced stress using electroencephalogram signals: A framework based on wearable safety-critical scheme and machine learning10.1016/j.inffus.2019.06.006 ↗
36. Research on work efficiency and light comfort based on EEG evaluation method10.1016/j.buildenv.2020.107122 ↗
37. Balan, O., Moise, G., Moldoveanu, A., Leordeanu, M., and Moldoveanu, F. (2019). Fear level classification based on emotional dimensions and machine learning techniques. Sensors, 19.10.3390/s19071738 ↗
38. Explainable artificial intelligence to evaluate industrial internal security using EEG signals in IoT framework10.1016/j.adhoc.2021.102641 ↗
39. Distinguishing and quantifying the visual aesthetics of a product: An integrated approach of eye-tracking and EEG10.1016/j.ergon.2019.02.006 ↗
40. Affect recognition from scalp-EEG using channel-wise encoder networks coupled with geometric deep learning and multi-channel feature fusion10.1016/j.knosys.2022.109038 ↗
41. Study of auditory brain cognition laws-based recognition method of automobile sound quality10.3389/fnhum.2021.663049 ↗
42. The induction and detection method of angry driving evidences from EEG and physiological signals10.1155/2018/3702795 ↗
43. Review of individual differences from transfer learning10.1134/S1019331622040189 ↗
44. Mohsen, S., and Alharbi, A.G. (2021, January 9–11). EEG-based human emotion prediction using an LSTM model. Proceedings of the 2021 IEEE International Midwest Symposium on Circuits and Systems, Lansing, MI, USA.10.1109/MWSCAS47672.2021.9531707 ↗
45. Stress in interactive applications: Analysis of the valence-arousal space based on physiological signals and self-reported data10.1007/s11042-016-3637-2 ↗
46. A wavelet-based approach to emotion classification using EDA signals10.1016/j.eswa.2018.06.014 ↗
47. Directing physiology and mood through music: Validation of an affective music player10.1109/T-AFFC.2012.28 ↗
48. Lin, W.Q., Li, C., and Zhang, Y.J. (2022). Interactive application of dataglove based on emotion recognition and judgment system. Sensors, 22.10.3390/s22176327 ↗
49. The tactile perception evaluation of wood surface with different roughness and shapes: A study using galvanic skin response10.37763/wr.1336-4561/67.2.311325 ↗
50. A multimodal framework for large-scale emotion recognition by fusing music and electrodermal activity signals10.1145/3490686 ↗
51. Romaniszyn-Kania, P., Pollak, A., Danch-Wierzchowska, M., Kania, D., Mysliwiec, A.P., Pietka, E., and Mitas, A.W. (2020). Hybrid system of emotion evaluation in physiotherapeutic procedures. Sensors, 21.10.3390/s20216343 ↗
52. Sepulveda, A., Castillo, F., Palma, C., and Rodriguez-Fernandez, M. (2021). Emotion recognition from ECG signals using wavelet scattering and machine learning. Appl. Sci., 11.10.3390/app11114945 ↗
53. Driver emotion recognition of multiple-ECG-feature fusion based on BP network and D-S evidence10.1049/iet-its.2019.0499 ↗
54. Evaluation of effect of music on human nervous system by heart rate variability analysis using ECG sensor
55. Physiological mouse: Toward an emotion-aware mouse10.1007/s10209-016-0469-9 ↗
56. Hu, J.Y., and Li, Y. (2022, January 1–3). Electrocardiograph based emotion recognition via WGAN-GP data enhancement and improved CNN. Proceedings of the 15th International Conference on Intelligent Robotics and Applications-Smart Robotics for Society, Intelligent Robotics and Applications, Harbin, China.10.1007/978-3-031-13844-7_16 ↗
57. Can you feel what you do not see? Using internal feedback to detect briefly presented emotional stimuli10.1016/j.ijpsycho.2011.04.007 ↗
58. Non-contact emotion recognition combining heart rate and facial expression for interactive gaming environments10.1109/ACCESS.2020.2964794 ↗
59. Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal10.1007/978-3-319-71589-6_33 ↗
60. Emotion visualization system based on physiological signals combined with the picture and scene10.1177/14738716221109146 ↗
61. Emotion classification on eye-tracking and electroencephalograph fused signals employing deep gradient neural networks10.1016/j.asoc.2021.107752 ↗
62. Tang, Z.C., Xia, D., Li, X.T., Wang, X.Y., Ying, J.C., and Yang, H.C. (2022). Evaluation of the effect of music on idea generation using electrocardiography and electroencephalography signals. Int. J. Technol. Des. Educ.10.1007/s10798-022-09782-x ↗
63. A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals10.1016/j.bspc.2013.06.014 ↗
64. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices10.1109/JBHI.2017.2688239 ↗
65. EMG and GSR signals for evaluating user’s perception of different types of ceramic flooring10.1016/j.ergon.2008.02.011 ↗
66. Emotion recognition from multimodal physiological signals using a regularized deep fusion deep fusion of kernel machine10.1109/TCYB.2020.2987575 ↗
67. Reliability of physiological responses induced by basic emotions: A pilot study10.1186/s40101-019-0209-y ↗
68. Emotion extraction based on multi bio-signal using back-propagation neural network10.1007/s11042-016-4213-5 ↗
69. Reliable emotion recognition system based on dynamic adaptive fusion of forehead bio-potentials and physiological signals10.1016/j.cmpb.2015.07.006 ↗
70. Emotion prediction from physiological signals: A comparison study between visual and auditory elicitors10.1093/iwc/iwt039 ↗
71. Emotion classification with multichannel physiological signals using hybrid feature and adaptive decision fusion10.1016/j.bspc.2021.103235 ↗
72. A multimodal database as a background for emotional synthesis, recognition and training in e-learning systems10.1007/11573548_73 ↗
73. Toward machine emotionalintelligence: Analysis of affective physiological state10.1109/34.954607 ↗
74. Physiological-based affect event detector for entertainment video applications10.1109/T-AFFC.2012.2 ↗
75. Emotion recognition system using short-term monitoring of physiological signals10.1007/BF02344719 ↗
76. The impact of system feedback on learners’ affective and physiological states10.1007/978-3-642-13388-6_31 ↗
77. A fuzzy psycho-physiological approach to enable the understanding of an engineer’s affect status during CAD activities10.1016/j.cad.2013.10.007 ↗
78. Pinto, G., Carvalho, J.M., Barros, F., Soares, S.C., Pinho, A.J., and Bras, S. (2020). Multimodal emotion evaluation: A physiological model for cost-effective emotion classification. Sensors, 20.10.3390/s20123510 ↗
79. Two-dimensional emotion evaluation with multiple physiological signals10.1007/978-3-319-94944-4_18 ↗
80. Machine learning model for mapping of music mood and human emotion based on physiological signals10.1007/s11042-021-11650-0 ↗
81. Real-time emotion recognition system with multiple physiological signals10.1299/jamdsm.2019jamdsm0075 ↗
82. Lin, W.Q., Li, C., and Zhang, Y.M. (2022). Model of emotion judgment based on features of multiple physiological signals. Appl. Sci., 12.10.3390/app12104998 ↗
83. iAware: A real-time emotional biofeedback system based on physiological signals10.1109/ACCESS.2018.2885279 ↗
84. Chen, S.H., Tang, J.J., Zhu, L., and Kong, W.Z. (2022). A multi-stage dynamical fusion network for multimodal emotion recognition. Cogn. Neurodynamics.10.1007/s11571-022-09851-w ↗
85. User experience evaluation in virtual reality based on subjective feelings and physiological signals10.2352/J.ImagingSci.Technol.2019.63.6.060413 ↗
86. A real-time emotionality assessment (RTEA) system based on psycho-physiological evaluation10.1080/00207450701769380 ↗
87. Uluer, P., Kose, H., Gumuslu, E., and Barkana, D.E. (2021). Experience with an affective robot assistant for children with hearing disabilities. Int. J. Soc. Robot.10.1007/s12369-021-00830-5 ↗
88. AIEFS and HEC based emotion estimation using physiological measurements for the children with autism spectrum disorder
89. Improving driving safety by detecting negative emotions with biological signals: Which is the best?10.1177/03611981211041594 ↗
90. Multi-modal physiological data fusion for affect estimation using deep learning10.1109/ACCESS.2021.3055933 ↗
91. Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers10.1016/j.eswa.2017.01.040 ↗
92. Saffaryazdi, N., Goonesekera, Y., Saffaryazdi, N., Hailemariam, N.D., Temesgen, E.G., Nanayakkara, S., Broadbent, E., and Billinghurst, M. (2022, January 22–25). Emotion recognition in conversations using brain and physiological signals. Proceedings of the 27th International Conference on Intelligent User Interfaces, University Helsinki, Electricity Network, Helsinki, Finland.10.1145/3490099.3511148 ↗
93. Ma, R.X., Yan, X., Liu, Y.Z., Li, H.L., and Lu, B.L. (2021, January 1–5). Sex difference in emotion recognition under sleep deprivation: Evidence from EEG and eye-tracking. Proceedings of the IEEE Engineering in Medicine and Biology Society Conference Proceedings, Virtual Conference.10.1109/EMBC46164.2021.9630808 ↗
94. Singson, L.N.B., Sanchez, M.T.U.R., and Villaverde, J.F. (2021, January 20–22). Emotion recognition using short-term analysis of heart rate variability and ResNet architecture. Proceedings of the 13th International Conference on Computer and Automation Engineering, Melbourne, VIC, Australia.10.1109/ICCAE51876.2021.9426094 ↗
95. Hinduja, S., Kaur, G., and Canavan, S. (October, January 28). Investigation into recognizing context over time using physiological signals. Proceedings of the 9th International Conference on Affective Computing and Intelligent Interaction (ACII), Nara, Japan.
96. Ethical issues surrounding human participants research using the Internet10.1207/S15327019EB1303_01 ↗
97. Cowie, R. (2015). The Oxford Handbook of Affective Computing (Oxford Library of Psychology), Oxford University Press.
98. Parallel circuits from the bed nucleiof stria terminalis to the lateral hypothalamus drive opposing emotional states10.1038/s41593-018-0198-x ↗
99. Review on stimuli presentation for affect analysis based on EEG10.1109/ACCESS.2020.2980893 ↗
100. Privacy-phys: Facial video-based physiological modification for privacy protection10.1109/LSP.2022.3185964 ↗