Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review
Source abstract
Abstract
Recent developments and studies in brain-computer interface (BCI) technologies have facilitated emotion detection and classification. Many BCI studies have sought to investigate, detect, and recognize participants’ emotional affective states. The applied domains for these studies are varied, and include such fields as communication, education, entertainment, and medicine. To understand trends in electroencephalography (EEG)-based emotion recognition system research and to provide practitioners and researchers with insights into and future directions for emotion recognition systems, this study set out to review published articles on emotion detection, recognition, and classification. The study also reviews current and future trends and discusses how these trends may impact researchers and practitioners alike. We reviewed 285 articles, of which 160 were refereed journal articles that were published since the inception of affective computing research. The articles were classified based on a scheme consisting of two categories: research orientation and domains/applications. Our results show considerable growth of EEG-based emotion detection journal publications. This growth reflects an increased research interest in EEG-based emotion detection as a salient and legitimate research area. Such factors as the proliferation of wireless EEG devices, advances in computational intelligence techniques, and machine learning spurred this growth.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: EEG and Brain-Computer Interfaces · Emotion and Mood Recognition · Neural dynamics and brain function
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer and Information Sciences, Imam Muhammad bin Saud University, Riyadh 11432, Saudi Arabia
College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia
Center for Complex Engineering Systems at KACST and MIT, King Abdulaziz City for Science and Technology, Riyadh 11442, Saudi Arabia
References (100+)
Show source-provided references
1. Brain-computer interaction research at the computer vision and multimedia laboratory, university of geneva10.1109/TNSRE.2006.875544 ↗
2. Using brain-computer interfaces to detect human satisfaction in human-robot interaction10.1142/S0219843611002356 ↗
3. Emotion and attention: Event-related brain potential studies10.1016/S0079-6123(06)56002-9 ↗
4. Chien, V.S.C., Tsai, A.C., Yang, H.H., Tseng, Y.L., Savostyanov, A.N., and Liou, M. (2016). Conscious and non-conscious representations of emotional faces in asperger’s syndrome. JOVE J. Vis. Exp.10.3791/53962 ↗
5. Event-related theta synchronization predicts deficit in facial affect recognition in schizophrenia10.1037/a0035793 ↗
6. An effective neurofeedback intervention to improve social interactions in children with autism spectrum disorder10.1007/s10803-015-2523-5 ↗
7. Extracting neurophysiological signals reflecting users’ emotional and affective responses to bci use: A systematic literature review10.3233/NRE-151266 ↗
8. Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals10.1016/j.neuroimage.2013.11.007 ↗
9. Culture in social neuroscience: A review10.1080/17470919.2012.695293 ↗
10. The mirror neuron system: New frontiers10.1080/17470910802408513 ↗
11. The development of the social brain in human infancy10.1111/j.1460-9568.2007.05379.x ↗
12. A dual mechanism neural framework for social understanding10.1080/09515080601085864 ↗
13. Alotaiby, T., Abd El-Samie, F.E., Alshebeili, S.A., and Ahmad, I. (2015). A review of channel selection algorithms for EEG signal processing. EURASIP J. Adv. Signal Process.10.1186/s13634-015-0251-9 ↗
14. Feature extraction and selection for emotion recognition from EEG10.1109/TAFFC.2014.2339834 ↗
15. Motivation, emotion, and their inhibitory control mirrored in brain oscillations10.1016/j.neubiorev.2006.10.004 ↗
16. Kim, M.K., Kim, M., Oh, E., and Kim, S.P. (2013). A review on the computational methods for emotional state estimation from the human EEG. Comput. Math. Methods Med., 2013.10.1155/2013/573734 ↗
17. Neural correlates of cognitive improvements following cognitive remediation in schizophrenia: A systematic review of randomized trials10.3402/snp.v6.30054 ↗
18. Neuroplastic changes following social cognition training in schizophrenia: A systematic review10.1007/s11065-016-9326-0 ↗
19. Finding a way in: A review and practical evaluation of fmri and EEG for detection and assessment in disorders of consciousness10.1016/j.neubiorev.2013.05.004 ↗
20. Computer-aided diagnosis of depression using EEG signals10.1159/000381950 ↗
21. Automated diagnosis of autism: In search of a mathematical marker10.1515/revneuro-2014-0036 ↗
22. Adaptation in affective video games: A literature review
23. Reyes-Munoz, A., Domingo, M.C., Lopez-Trinidad, M.A., and Delgado, J.L. (2016). Integration of body sensor networks and vehicular ad-hoc networks for traffic safety. Sensors, 16.10.3390/s16010107 ↗
24. EEG-neurofeedback for optimising performance. I: A review of cognitive and affective outcome in healthy participants10.1016/j.neubiorev.2013.09.015 ↗
25. Action-based model of dissonance: A review, integration, and expansion of conceptions of cognitive conflict
26. Subject-independent odor pleasantness classification using brain and peripheral signals10.1109/TAFFC.2015.2496310 ↗
27. Keuper, K., Zwitserlood, P., Rehbein, M.A., Eden, A.S., Laeger, I., Junghofer, M., Zwanzger, P., and Dobel, C. (2013). Early prefrontal brain responses to the hedonic quality of emotional words—A simultaneous EEG and MEG study. PLoS ONE, 8.10.1371/journal.pone.0070788 ↗
28. Motivational mechanisms (bas) and prefrontal cortical activation contribute to recognition memory for emotional words. Rtms effect on performance and EEG (alpha band) measures10.1016/j.bandl.2014.08.006 ↗
29. Emotion word recognition: Discrete information effects first, continuous later?10.1016/j.brainres.2014.03.045 ↗
30. On the roles of distinctiveness and semantic expectancies in episodic encoding of emotional words10.1111/psyp.12537 ↗
31. Individual differences in emotion word processing: A diffusion model analysis10.3758/s13415-016-0408-5 ↗
32. Imbir, K.K., Spustek, T., and Zygierewicz, J. (2016). Effects of valence and origin of emotions in word processing evidenced by event related potential correlates in a lexical decision task. Front. Psychol., 7.10.3389/fpsyg.2016.00271 ↗
33. Motivational processing of food cues in anorexia nervosa: A pilot study10.1007/s40519-014-0114-7 ↗
34. Emotion assessment from physiological signals for adaptation of game difficulty10.1109/TSMCA.2011.2116000 ↗
35. Tzieropoulos, H., de Peralta, R.G., Bossaerts, P., and Andino, S.L.G. (2011). The impact of disappointment in decision making: Inter-individual differences and electrical neuroimaging. Front. Hum. Neurosci., 4.10.3389/fnhum.2010.00235 ↗
36. Spape, M.M., Kivikangas, J.M., Jarvela, S., Kosunen, I., Jacucci, G., and Ravaja, N. (2013). Keep your opponents close: Social context affects EEG and femg linkage in a turn-based computer game. PLoS ONE, 8.10.1371/journal.pone.0078795 ↗
37. The interplay between feedback-related negativity and individual differences in altruistic punishment: An EEG study10.3758/s13415-015-0388-x ↗
38. Charland, P., Leger, P.M., Senecal, S., Courtemanche, F., Mercier, J., Skelling, Y., and Labonte-Lemoyne, E. (2015). Assessing the multiple dimensions of engagement to characterize learning: A neurophysiological perspective. JOVE J. Vis. Exp.10.3791/52627 ↗
39. Lopez-Gil, J.M., Virgili-Goma, J., Gil, R., and Garcia, R. (2016). Method for improving EEG based emotion recognition by combining it with synchronized biometric and eye tracking technologies in a non-invasive and low cost way. Front. Comput. Neurosci., 10.10.3389/fncom.2016.00085 ↗
40. Abdur-Rahim, J., Morales, Y., Gupta, P., Umata, I., Watanabe, A., Even, J., Suyama, T., and Ishii, S. (2016). Multi-sensor based state prediction for personal mobility vehicles. PLoS ONE, 11.10.1371/journal.pone.0162593 ↗
41. Multi-modal emotion analysis from facial expressions and electroencephalogram10.1016/j.cviu.2015.09.015 ↗
42. Construction and cross-correlation analysis of the affective physiological response database10.1007/s11432-010-4001-1 ↗
43. Development in the neurophysiology of emotion processing and memory in school-age children10.1016/j.dcn.2014.07.007 ↗
44. Daly, I., Williams, D., Kirke, A., Weaver, J., Malik, A., Hwang, F., Miranda, E., and Nasuto, S.J. (2016). Affective brain-computer music interfacing. J. Neural Eng., 13.10.1088/1741-2560/13/4/046022 ↗
45. An investigation into the use of six facially encoded emotions in brain-computer interfacing10.1080/2326263X.2016.1149360 ↗
46. Gender differences in implicit and explicit processing of emotional facial expressions as revealed by event-related theta synchronization10.1037/a0019175 ↗
47. Emotion processing in Parkinson’s disease: Dissociation between early neuronal processing and explicit ratings10.1016/j.clinph.2005.09.009 ↗
48. Electrophysiological correlates of remembering emotional pictures10.1016/j.neuroimage.2010.07.030 ↗
49. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks10.1109/TAMD.2015.2431497 ↗
50. An unsupervised discriminative extreme learning machine and its applications to data clustering10.1016/j.neucom.2014.11.097 ↗
51. The effects of emotional intensity on ERP correlates of recognition memory10.1097/WNR.0b013e3283229b52 ↗
52. Lu, X.J., Ho, H.T., Liu, F., Wu, D.X., and Thompson, W.F. (2015). Intonation processing deficits of emotional words among mandarin chinese speakers with congenital amusia: An erp study. Front. Psychol., 6.10.3389/fpsyg.2015.00385 ↗
53. Lin, Y.P., Yang, Y.H., and Jung, T.P. (2014). Fusion of electroencephalographic dynamics and musical contents for estimating emotional responses in music listening. Front. Neurosci., 8.10.3389/fnins.2014.00094 ↗
54. Facial emotion recognition in schizophrenia: When and why does it go awry?10.1016/j.schres.2007.05.001 ↗
55. Chen, X.H., Yang, J.F., Gan, S.Z., and Yang, Y.F. (2012). The contribution of sound intensity in vocal emotion perception: Behavioral and electrophysiological evidence. PLoS ONE, 7.10.1371/journal.pone.0030278 ↗
56. Oscillatory brain dynamics associated with the automatic processing of emotion in words10.1016/j.bandl.2014.07.011 ↗
57. Emotional state classification from EEG data using machine learning approach10.1016/j.neucom.2013.06.046 ↗
58. Lin, Y.P., Duann, J.R., Feng, W.F., Chen, J.H., and Jung, T.P. (2014). Revealing spatio-spectral electroencephalographic dynamics of musical mode and tempo perception by independent component analysis. J. Neuroeng. Rehabil., 11.10.1186/1743-0003-11-18 ↗
59. Distraction reduces theta synchronization in emotion regulation during adolescence10.1016/j.neulet.2013.05.070 ↗
60. Recognition advantage of happy faces: Tracing the neurocognitive processes10.1016/j.neuropsychologia.2013.07.010 ↗
61. Emotion recognition from single-trial EEG based on kernel fisher’s emotion pattern and imbalanced quasiconformal kernel support vector machine10.3390/s140813361 ↗
62. Neural processing of facial expressions of emotion in first onset psychosis10.1016/j.psychres.2014.06.017 ↗
63. Randomly dividing homologous samples leads to overinflated accuracies for emotion recognition10.1016/j.ijpsycho.2015.02.023 ↗
64. The relation between electroencephalogram asymmetry and attention biases to threat at baseline and under stress10.1016/j.bandc.2013.05.009 ↗
65. Liu, T.R., Xiao, T., and Shi, J.N. (2016). Automatic change detection to facial expressions in adolescents: Evidence from visual mismatch negativity responses. Front. Psychol., 7.10.3389/fpsyg.2016.00462 ↗
66. Positive and negative symptom scores are correlated with activation in different brain regions during facial emotion perception in schizophrenia patients: A voxel-based sloreta source activity study10.1016/j.schres.2013.10.025 ↗
67. Lin, H.Y., Xiang, J., Li, S.L., Liang, J.F., and Jin, H. (2015). Anticipation of negative pictures enhances the p2 and p3 in their later recognition. Front. Hum. Neurosci., 9.10.3389/fnhum.2015.00646 ↗
68. Zhang, D.D., Wang, L.L., Luo, Y., and Luo, Y.J. (2012). Individual differences in detecting rapidly presented fearful faces. PLoS ONE, 7.10.1371/journal.pone.0049517 ↗
69. Yu, B., Ma, L., Li, H.F., Zhao, L., Bo, H.J., and Wang, X.D. (2016). Biological computation indexes of brain oscillations in unattended facial expression processing based on event-related synchronization/desynchronization. Comput. Math. Methods Med.10.1155/2016/8958750 ↗
70. Event-related synchronization of delta and beta oscillations reflects developmental changes in the processing of affective pictures during adolescence10.1016/j.ijpsycho.2013.10.005 ↗
71. Hybrid video emotional tagging using users' EEG and video content10.1007/s11042-013-1450-8 ↗
72. Emotion-elicited gamma synchrony in patients with first-episode schizophrenia: A neural correlate of social cognition outcomes
73. No evidence for mirror system dysfunction in schizophrenia from a multimodal tms/EEG study10.1016/j.psychres.2015.05.067 ↗
74. Shen, X.B., Wu, Q., Zhao, K., and Fu, X.L. (2016). Electrophysiological evidence reveals differences between the recognition of microexpressions and macroexpressions. Front. Psychol., 7.10.3389/fpsyg.2016.01346 ↗
75. Affective-motivational brain responses to direct gaze in children with autism spectrum disorder10.1111/j.1469-7610.2011.02522.x ↗
76. Emotion perception and electrophysiological correlates in Huntington’s disease10.1016/j.clinph.2013.12.111 ↗
77. Brain signatures of perceiving a smile: Time course and source localization10.1002/hbm.22917 ↗
78. Lin, H.Y., Schulz, C., and Straube, T. (2015). Cognitive tasks during expectation affect the congruency ERP effects to facial expressions. Front. Hum. Neurosci., 9.10.3389/fnhum.2015.00596 ↗
79. Effects of subcallosal cingulate deep brain stimulation on negative self-bias in patients with treatment-resistant depression10.1016/j.brs.2014.11.010 ↗
80. Symmetry perception and affective responses: A combined EEG/emg study10.1016/j.neuropsychologia.2012.10.003 ↗
81. A typical processing of fearful face-voice pairs in pervasive developmental disorder: An ERP study10.1016/j.clinph.2008.05.005 ↗
82. Detection of concealed information by p3 and frontal EEG asymmetry10.1016/j.neulet.2013.01.029 ↗
83. Lin, H.Y., Schulz, C., and Straube, T. (2015). Fearful contextual expression impairs the encoding and recognition of target faces: An ERP study. Front. Behav. Neurosci., 9.10.3389/fnbeh.2015.00237 ↗
84. The late positive potential: A neurophysiological marker for emotion regulation in children10.1111/j.1469-7610.2009.02168.x ↗
85. The influence of color on emotional perception of natural scenes10.1111/j.1469-8986.2011.01284.x ↗
86. Neural modulation of directed forgetting by valence and arousal: An event-related potential study10.1016/j.brainres.2016.08.009 ↗
87. The effects of aging on emotion-induced modulations of source retrieval ERPS: Evidence for valence biases10.1016/j.neuropsychologia.2012.09.024 ↗
88. Multimodal emotion recognition in response to videos10.1109/T-AFFC.2011.37 ↗
89. Processing of prosodic changes in natural speech stimuli in school-age children10.1016/j.ijpsycho.2012.09.010 ↗
90. Fearful face recognition in schizophrenia: An electrophysiological study10.1016/j.schres.2013.06.044 ↗
91. Simultaneous recording of EEG and facial muscle reactions during spontaneous emotional mimicry10.1016/j.neuropsychologia.2007.10.019 ↗
92. Deap: A database for emotion analysis using physiological signals10.1109/T-AFFC.2011.15 ↗
93. Emotion processing in words: A test of the neural re-use hypothesis using surface and intracranial EEG10.1093/scan/nst034 ↗
94. Csukly, G., Stefanics, G., Komlosi, S., Czigler, I., and Czobor, P. (2013). Emotion-related visual mismatch responses in schizophrenia: Impairments and correlations with emotion recognition. PLoS ONE, 8.10.1371/journal.pone.0075444 ↗
95. Bhushan, V., Saha, G., Lindsen, J., Shimojo, S., and Bhattacharya, J. (2012). How we choose one over another: Predicting trial-by-trial preference decision. PLoS ONE, 7.10.1371/journal.pone.0043351 ↗
96. Danger and usefulness are detected early in auditory lexical processing: Evidence from electroencephalography10.1016/j.bandl.2012.05.005 ↗
97. What grabs his attention but not hers? Estrogen correlates with neurophysiological measures of vocal change detection10.1016/j.psyneuen.2008.02.010 ↗
98. Deficits in low beta desynchronization reflect impaired emotional processing in schizophrenia10.1016/j.schres.2016.01.031 ↗
99. Too bad: Bias for angry faces in social anxiety interferes with identity processing10.1016/j.neuropsychologia.2016.02.005 ↗
100. What strikes the strings of your heart?-multi-label dimensionality reduction for music emotion analysis via brain imaging10.1109/TAMD.2015.2429580 ↗