journal article·2021-05-27Open access

Emotion Recognition from ECG Signals Using Wavelet Scattering and Machine Learning

Axel Sepúlveda, Francisco Castillo, Carlos Palma, Maria Rodriguez-Fernandez

Applied Sciences· Volume 11· Issue 11

Abstract

Affect detection combined with a system that dynamically responds to a person’s emotional state allows an improved user experience with computers, systems, and environments and has a wide range of applications, including entertainment and health care. Previous studies on this topic have used a variety of machine learning algorithms and inputs such as audial, visual, or physiological signals. Recently, a lot of interest has been focused on the last, as speech or video recording is impractical for some applications. Therefore, there is a need to create Human–Computer Interface Systems capable of recognizing emotional states from noninvasive and nonintrusive physiological signals. Typically, the recognition task is carried out from electroencephalogram (EEG) signals, obtaining good accuracy. However, EEGs are difficult to register without interfering with daily activities, and recent studies have shown that it is possible to use electrocardiogram (ECG) signals for this purpose. This work improves the performance of emotion recognition from ECG signals using wavelet transform for signal analysis. Features of the ECG signal are extracted from the AMIGOS database using a wavelet scattering algorithm that allows obtaining features of the signal at different time scales, which are then used as inputs for different classifiers to evaluate their performance. The results show that the proposed algorithm for extracting features and classifying the signals obtains an accuracy of 88.8% in the valence dimension, 90.2% in arousal, and 95.3% in a two-dimensional classification, which is better than the performance reported in previous studies. This algorithm is expected to be useful for classifying emotions using wearable devices.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: EEG and Brain-Computer Interfaces · Emotion and Mood Recognition · ECG Monitoring and Analysis

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Institute for Biological and Medical Engineering, Schools of Engineering, Medicine and Biological Sciences, Pontificia Universidad Católica de Chile, Santiago 7820436, Chile

ATCAS-Grupo CLER, Temuco 4781136, Chile

Funding

Fondo Nacional de Desarrollo Científico, Tecnológico y de Innovación Tecnológica

References (57)

Show source-provided references
1. A circumplex model of affect10.1037/h0077714 ↗
2. Basu, S., Jana, N., Bag, A., Mahadevappa, M., Mukherjee, J., Kumar, S., and Guha, R. (2015, January 21–24). Emotion recognition based on physiological signals using valence-arousal model. Proceedings of the 2015 Third International Conference on Image Information Processing (ICIIP), Waknaghat, India.10.1109/ICIIP.2015.7414739 ↗
3. Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., and Yang, X. (2018). A Review of Emotion Recognition Using Physiological Signals. Sensors, 18.10.3390/s18072074 ↗
4. An accurate emotion recognition system using ECG and GSR signals and matching pursuit method10.1016/j.bj.2017.11.001 ↗
5. DEAP: A Database for Emotion Analysis Using Physiological Signals10.1109/T-AFFC.2011.15 ↗
6. A Multi-Modal Affective Database for Affect Recognition and Implicit Tagging10.1109/T-AFFC.2011.25 ↗
7. Miranda Correa, J.A., Abadi, M.K., Sebe, N., and Patras, I. (2018). AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups. IEEE Trans. Affect. Comput.
8. EmoNets: Multimodal deep learning approaches for emotion recognition in video10.1007/s12193-015-0195-2 ↗
9. AFEW-VA database for valence and arousal estimation in-the-wild10.1016/j.imavis.2017.02.001 ↗
10. Dhall, A., Goecke, R., Joshi, J., Hoey, J., and Gedeon, T. (2016, January 12–16). Emotiw 2016: Video and group-level emotion recognition challenges. Proceedings of the 18th ACM international conference on multimodal interaction, Tokyo, Japan.10.1145/2993148.2997638 ↗
11. Ranganathan, H., Chakraborty, S., and Panchanathan, S. (2016, January 7–10). Multimodal emotion recognition using deep learning architectures. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477679 ↗
12. Fan, Y., Lu, X., Li, D., and Liu, Y. (2016, January 12–16). Video-based emotion recognition using CNN-RNN and C3D hybrid networks. Proceedings of the 18th ACM International Conference on Multimodal Interaction-ICMI 2016, Tokyo, Japan.10.1145/2993148.2997632 ↗
13. Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks10.1016/j.jvcir.2018.12.039 ↗
14. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The extended Cohn-Kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
15. Valstar, M., and Pantic, M. (2010, January 3–6). Induced disgust, happiness and surprise: An addition to the mmi facial expression database. Proceedings of the 3rd International Workshop on EMOTION (Satellite of LREC): Corpora for Research on Emotion and Affect, Paris, France.
16. Zheng, W.L., Dong, B.N., and Lu, B.L. (2014, January 26–30). Multimodal emotion recognition using EEG and eye tracking data. Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA.
17. Emotion Recognition based on EEG using LSTM Recurrent Neural Network
18. Bălan, O., Moise, G., Petrescu, L., Moldoveanu, A., Leordeanu, M., and Moldoveanu, F. (2020). Emotion Classification Based on Biophysical Signals and Machine Learning Techniques. Symmetry, 12.10.3390/sym12010021 ↗
19. Paszkiel, S. (2017). Characteristics of question of blind source separation using Moore-Penrose pseudoinversion for reconstruction of EEG signal. International Conference on Automation, Springer.10.1007/978-3-319-54042-9_36 ↗
20. Automatic ECG-Based Emotion Recognition in Music Listening10.1109/TAFFC.2017.2781732 ↗
21. Chicco, D., and Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom., 21.10.1186/s12864-019-6413-7 ↗
22. DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices10.1109/JBHI.2017.2688239 ↗
23. Harper, R., and Southern, J. (2020). A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeat. IEEE Trans. Affect. Comput.
24. Ferdinando, H., Seppänen, T., and Alasaarela, E. (2016, January 5–7). Comparing features from ECG pattern and HRV analysis for emotion recognition system. Proceedings of the 2016 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), Chiang Mai, Thailand.10.1109/CIBCB.2016.7758108 ↗
25. Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS)10.1109/ACCESS.2018.2883213 ↗
26. Sarkar, P., and Etemad, A. (2020, January 4–8). Self-Supervised Learning for ECG-Based Emotion Recognition. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053985 ↗
27. Tung, K., Liu, P.K., Chuang, Y.C., Wang, S.H., and Wu, A.Y. (2018, January 3–6). Entropy-Assisted Multi-Modal Emotion Recognition Framework Based on Physiological Signals. Proceedings of the 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), Sarawak, Malaysia.10.1109/IECBES.2018.8626634 ↗
28. An affective computing approach to physiological emotion specificity: Toward subject-independent and stimulus-independent classification of film-induced emotions10.1111/j.1469-8986.2010.01170.x ↗
29. Sarkar, P., and Etemad, A. (2020). Self-supervised ECG Representation Learning for Emotion Recognition. IEEE Trans. Affect. Comput.
30. Siddharth, S., Jung, T., and Sejnowski, T.J. (2019). Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing. IEEE Trans. Affect. Comput.
31. Statistical notes for clinical researchers: Chi-squared test and Fisher’s exact test10.5395/rde.2017.42.2.152 ↗
32. Multi-objective optimization for personalized prediction of venous thromboembolism in ovarian cancer patients
33. Handling imbalanced datasets: A review
34. Filtering Poincareplots10.12921/cmst.2005.11.01.39-48 ↗
35. Luo, D., Pan, W., Li, Y., Feng, K., and Liu, G. (2018). The interaction analysis between the sympathetic and parasympathetic systems in CHF by using transfer entropy method. Entropy, 20.10.3390/e20100795 ↗
36. Strüven, A., Holzapfel, C., Stremmel, C., and Brunner, S. (2021). Obesity, Nutrition and Heart Rate Variability. Int. J. Mol. Sci., 22.10.3390/ijms22084215 ↗
37. Ten Lectures on Wavelets10.1063/1.4823127 ↗
38. Group Invariant Scattering10.1002/cpa.21413 ↗
39. Deep Scattering Spectrum10.1109/TSP.2014.2326991 ↗
40. Invariant Scattering Convolution Networks10.1109/TPAMI.2012.230 ↗
41. A theory for multiresolution signal decomposition: The wavelet representation10.1109/34.192463 ↗
42. James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning, Springer.10.1007/978-1-4614-7138-7 ↗
43. Linear discriminant analysis: A detailed tutorial10.3233/AIC-170729 ↗
44. Vishwanath, M., Jafarlou, S., Shin, I., Lim, M.M., Dutt, N., Rahmani, A.M., and Cao, H. (2020). Investigation of machine learning approaches for traumatic brain injury classification via EEG assessment in mice. Sensors, 20.10.3390/s20072027 ↗
45. Khanna, D., and Sharma, A. (2018). Kernel-based naive bayes classifier for medical predictions. Intelligent Engineering Informatics, Springer.10.1007/978-981-10-7566-7_10 ↗
46. Learning k for knn classification
47. High dimensional data classification and feature selection using support vector machines10.1016/j.ejor.2017.08.040 ↗
48. Truong, X.L., Mitamura, M., Kono, Y., Raghavan, V., Yonezawa, G., Truong, X.Q., Do, T.H., Tien Bui, D., and Lee, S. (2018). Enhancing prediction performance of landslide susceptibility model using hybrid machine learning approach of bagging ensemble and logistic model tree. Appl. Sci., 8.10.3390/app8071046 ↗
49. Reliable accuracy estimates from k-fold cross validation10.1109/TKDE.2019.2912815 ↗
50. The need to approximate the use-case in clinical machine learning10.1093/gigascience/gix019 ↗
51. Deep Neural Networks for Human Activity Recognition With Wearable Sensors: Leave-One-Subject-Out Cross-Validation for Model Selection10.1109/ACCESS.2020.3010715 ↗
52. Wavelet Scattering Transform for ECG Beat Classification10.1155/2020/3215681 ↗
53. Emotion Recognition by Heart Rate Variability
54. ECG signal classification using ensemble decision tree
55. Paszkiel, S., Hunek, W., and Shylenko, A. (2016). Project and Simulation of a Portable Device for Measuring Bioelectrical Signals from the Brain for States Consciousness Verification with Visualization on LEDs. International Conference on Automation, Springer.10.1007/978-3-319-29357-8_3 ↗
56. Application of wavelet analysis in EMG feature extraction for pattern classification10.2478/v10048-011-0009-y ↗
57. Arozi, M., Caesarendra, W., Ariyanto, M., Munadi, M., Setiawan, J.D., and Glowacz, A. (2020). Pattern recognition of single-channel sEMG signal using PCA and ANN method to classify nine hand movements. Symmetry, 12.10.3390/sym12040541 ↗