Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks
Source abstract
Abstract
The aim of this study is to recognize human emotions by electroencephalographic (EEG) signals. The innovation of our research methods involves two aspects: First, we integrate the spatial characteristics, frequency domain, and temporal characteristics of the EEG signals, and map them to a two-dimensional image. With these images, we build a series of EEG Multidimensional Feature Image (EEG MFI) sequences to represent the emotion variation with EEG signals. Second, we construct a hybrid deep neural network to deal with the EEG MFI sequences to recognize human emotional states where the hybrid deep neural network combined the Convolution Neural Networks (CNN) and Long Short-Term-Memory (LSTM) Recurrent Neural Networks (RNN). Empirical research is carried out with the open-source dataset DEAP (a Dataset for Emotion Analysis using EEG, Physiological, and video signals) using our method, and the results demonstrate the significant improvements over current state-of-the-art approaches in this field. The average emotion classification accuracy of each subject with CLRNN (the hybrid neural networks that we proposed in this study) is 75.21%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: EEG and Brain-Computer Interfaces · Emotion and Mood Recognition · Gaze Tracking and Assistive Technology
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of International WIC, Beijing University of Technology, Beijing 100124, China
Beijing Advanced Innovation Center for Future Internet Technology, Beijing University of Technology, Beijing 100124, China
Knowledge Information Systems Lab, Department of Life Science and Informatics, Maebashi Institute of Technology, Maebashi 371-0816, Japan
Funding
National Basic Research Program of China · National Natural Science Foundation of China · International Science & Technology Cooperation Program of China
References (52)
Show source-provided references
1. Using psychophysiological techniques to measure user experience with entertainment technologies10.1080/01449290500331156 ↗
2. Detecting stress during real-world driving tasks using physiological sensors10.1109/TITS.2005.848368 ↗
3. Toward emotion recognition in car-racing drivers: A biosignal processing approach10.1109/TSMCA.2008.918624 ↗
4. An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders10.1016/j.bspc.2010.12.001 ↗
5. Verschuere, B., Ben-Shakhar, G.M., and Meijer, E. (2011). Memory Detection: Theory and Application of the Concealed Information Test. Psychopathy and the Detection of Concealed Information, Cambridge University Press.10.1017/CBO9780511975196 ↗
6. Toward machine emotional intelligence: Analysis of affective physiological state10.1109/34.954607 ↗
7. Survey on speech emotion recognition: Features, classification schemes, and databases10.1016/j.patcog.2010.09.020 ↗
8. On the simultaneous recognition of identity and expression from BU-3DFE datasets10.1016/j.patrec.2012.05.015 ↗
9. What does your chair know about your stress level?10.1109/TITB.2009.2035498 ↗
10. Lewis, M., and Haviland-Jones, J.M. (2000). The psychophysiology of emotion. Handbook of Emotion, Guilford Press.
11. Deap: A database for emotion analysis ;using physiological signals10.1109/T-AFFC.2011.15 ↗
12. A review on the computational methods for emotional state estimation from the human EEG
13. Ansari-Asl, K., Chanel, G., and Pun, T. (2007, January 3–7). A channel selection method for EEG classification in emotion assessment based on synchronization likelihood. Proceedings of the 15th European Signal Processing Conference, Poznan, Poland.
14. Horlings, R., Datcu, D., and Rothkrantz, L.J.M. (2008, January 12–13). Emotion recognition using brain activity. Proceedings of the 9th international conference on computer systems and technologies and workshop for PhD students in computing, Gabrovo, Bulgaria.10.1145/1500879.1500888 ↗
15. Classification of human emotion from EEG using discrete wavelet transform10.4236/jbise.2010.34054 ↗
16. Emotion recognition from EEG using higher order crossings10.1109/TITB.2009.2034649 ↗
17. Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis10.1109/T-AFFC.2010.7 ↗
18. Khalili, Z., and Moradi, M.H. (2008, January 18–20). Emotion detection using brain and peripheral signals. Proceedings of the Biomedical Engineering Conference, Cairo, Egypt.10.1109/CIBEC.2008.4786096 ↗
19. Mu, L., and Lu, B.-L. (2009, January 3–6). Emotion classification based on gamma-band EEG. Proceedings of the Annual International Conference of the IEEE, Minneapolis, MN, USA.10.1109/IEMBS.2009.5334139 ↗
20. Liu, Y., and Sourina, O. (2012, January 9–13). EEG-based dominance level recognition for emotion-enabled interaction. Proceedings of the IEEE International Conference on Multimedia and Expo, Melbourne, Australia.10.1109/ICME.2012.20 ↗
21. Rozgic, V., Vitaladevuni, S.N., and Prasad, R. (2013, January 26–31). Robust EEG emotion classification using segment level decision fusion. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6637858 ↗
22. Daunizeau, J., Lee, Y.-Y., and Hsieh, S. (2014). Classifying different emotional states by means of EEG-based functional connectivity patterns. PLoS ONE, 9.10.1371/journal.pone.0095415 ↗
23. Bashivan, P., Rish, I., Yeasin, M., and Codella, N. (2016, January 2–4). Learning representations from EEG with deep recurrent-convolutional neural networks. Proceedings of the International Conference on Learning Representations 2016, San Juan, PR, USA.
24. Cross-subject EEG feature selection for emotion recognition using transfer recursive feature elimination10.3389/fnbot.2017.00019 ↗
25. Chanel, G., Karim, A.-A., and Thierry, P. (2007, January 7–10). Valence-arousal evaluation using physiological signals in an emotion recall paradigm. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Montreal, QC, Canada.10.1109/ICSMC.2007.4413638 ↗
26. Nie, D., Wang, X.-W., Shi, L.-C., and Lu, B.-L. (May, January 27). EEG-based emotion recognition during watching movies. Proceedings of the 5th International IEEE/EMBS Conference on Neural Engineering, Cancun, Mexico.
27. Zheng, W.L., Dong, B.N., and Lu, B.-L. (2014, January 26–30). Multimodal emotion recognition using EEG and eye tracking data. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA.
28. Identifying stable patterns over time for emotion recognition from EEG
29. Continuous music-emotion recognition based on electroencephalogram10.1587/transinf.2015EDP7251 ↗
30. Inferring of human emotional states using multichannel EEG
31. Kroupi, E., Yazdani, A., and Ebrahimi, T. (2011, January 9–12). EEG correlates of different emotional states elicited during watching music videos. Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, Memphis, TN, USA.10.1007/978-3-642-24571-8_58 ↗
32. Toward an EEG-based recognition of music liking using time-frequency analysis10.1109/TBME.2012.2217495 ↗
33. Valence, arousal and dominance in the EEG during game play10.1504/IJAACS.2013.050691 ↗
34. An approach to EEG based emotion recognition and classification using kernel density estimation10.1016/j.procs.2015.04.138 ↗
35. Paul, S., Mazumder, A., Ghosh, P., Tibarewala, D.N., and Vimalarani, G. (2015, January 18–20). EEG based emotion recognition system using MFDFA as feature extractor. Proceedings of the International Conference on Robotics, Automation, Control and Embedded Systems (RACE), Chennai, India.10.1109/RACE.2015.7097247 ↗
36. An improved multi-scale entropy algorithm in emotion EEG features extraction10.1166/jmihi.2017.2031 ↗
37. Soleymani, M., Koelstra, S., Patras, I., and Pun, T. (2011, January 21–25). Continuous emotion detection in response to music videos. Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, Santa Barbara, CA, USA.10.1109/FG.2011.5771352 ↗
38. The ten-twenty electrode system of the international federation
39. Brown, L., Grundlehner, B., and Penders, J. (September, January 30). Towards wireless emotional valence detection from EEG. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA.
40. Toward emotion aware computing: An integrated approach using multichannel neurophysiological recordings and affective visual stimuli10.1109/TITB.2010.2041553 ↗
41. Schaaff, K., and Schultz, T. (2009, January 10–12). Towards emotion recognition from electroencephalographic signals. Proceedings of the 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, The Netherlands.10.1109/ACII.2009.5349316 ↗
42. Hosseini, S.A., Khalilzadeh, M.A., Naghibi-Sistani, M.B., and Niazmand, V. (2010, January 24–25). Higher order spectra analysis of EEG signals in emotional stress states. Proceedings of the Second International Conference on Information Technology and Computer Science, Kiev, Ukraine.10.1109/ITCS.2010.21 ↗
43. Chung, S.Y., and Yoon, H.J. (2012, January 17–21). Affective classification using Bayesian classifier and supervised learning. Proceedings of the 12th International Conference on Control, Automation and Systems, JeJu Island, Korea.
44. Li, X., Song, D., Zhang, P., Yu, G., Hou, Y., and Hu, B. (2016, January 15–18). Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Shenzhen, China.10.1109/BIBM.2016.7822545 ↗
45. Candra, H., Yuwono, M., Rifai, C., Handojoseno, A., Elamvazuthi, I., Nguyen, H.T., and Su, S. (2015, January 25–29). Investigation of window size in classification of EEG-emotion signal with wavelet entropy and support vector machine. Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy.10.1109/EMBC.2015.7320065 ↗
46. (2017, September 10). 10–20 System (EEG). Available online: https://en.wikipedia.org/wiki/10-20_system_(EEG).
47. Learning long-term dependencies with gradient descent is difficult10.1109/72.279181 ↗
48. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
49. (2017, September 10). DEEPLEARNING4J. Available online: https://deeplearning4j.org/.
50. A role for affective neuroscience in understanding stress: The case of separation distress circuitry10.1007/978-94-009-1990-7_4 ↗
51. A proposed mechanism of emotion10.1001/archneurpsyc.1937.02260220069003 ↗
52. Affective neuroscience: The emergence of a discipline10.1016/0959-4388(95)80029-8 ↗