Electroglottograph-Based Speech Emotion Recognition via Cross-Modal Distillation
Source abstract
Abstract
Speech emotion recognition (SER) is an important component of emotion computation and signal processing. Recently, many works have applied abundant acoustic features and complex model architectures to enhance the model’s performance, but these works sacrifice the portability of the model. To address this problem, we propose a model utilizing only the fundamental frequency from electroglottograph (EGG) signals. EGG signals are a sort of physiological signal that can directly reflect the movement of the vocal cord. Under the assumption that different acoustic features share similar representations in the internal emotional state, we propose cross-modal emotion distillation (CMED) to train the EGG-based SER model by transferring robust speech emotion representations from the log-Mel-spectrogram-based model. Utilizing the cross-modal emotion distillation, we achieve an increase of recognition accuracy from 58.98% to 66.80% on the S70 subset of the Chinese Dual-mode Emotional Speech Database (CDESD 7-classes) and 32.29% to 42.71% on the EMO-DB (7-classes) dataset, which shows that our proposed method achieves a comparable result with the human subjective experiment and realizes a trade-off between model complexity and performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Speech and Audio Processing · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic and Information Engineering, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing 100191, China
Funding
National Science Foundation for Young Scientists of China · National Natural Science Foundation of China · Fundamental Research Funds for the Central Universities
References (55)
Show source-provided references
1. Emotion recognition in human-computer interaction10.1109/79.911197 ↗
2. Ringeval, F., Michaud, A., Cifti, E., Güle, H., and Lalanne, D. (2018, January 22). AVEC 2018 Workshop and Challenge: Bipolar Disorder and Cross-Cultural Affect Recognition. Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop, Seoul, Korea.10.1145/3266302.3266316 ↗
3. Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M.A., Schuller, B., and Zafeiriou, S. (2016, January 20–25). Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472669 ↗
4. Neumann, M., and Vu, N.T. (2017, January 20–24). Attentive Convolutional Neural Network Based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech. Proceedings of the Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden.10.21437/Interspeech.2017-917 ↗
5. Kim, J., Englebienne, G., Truong, K.P., and Evers, V. (2017, January 23–27). Deep Temporal Models using Identity Skip-Connections for Speech Emotion Recognition. Proceedings of the 2017 ACM on Multimedia Conference, Mountain View, CA, USA.10.1145/3123266.3123353 ↗
6. Han, W., Ruan, H., Chen, X., Wang, Z., Li, H., and Schuller, B. (2018, January 2–6). Towards Temporal Modelling of Categorical Speech Emotion Recognition. Proceedings of the Interspeech 2018, Hyderabad, India.10.21437/Interspeech.2018-1858 ↗
7. Atmaja, B.T., and Akagi, M. (2019, January 16–18). Speech Emotion Recognition Based on Speech Segment Using LSTM with Attention Model. Proceedings of the 2019 IEEE International Conference on Signals and Systems (ICSigSys), Bandung, Indonesia.10.1109/ICSIGSYS.2019.8811080 ↗
8. Rajamani, S.T., Rajamani, K.T., Mallol-Ragolta, A., Liu, S., and Schuller, B. (2021, January 6–11). A Novel Attention-Based Gated Recurrent Unit and its Efficacy in Speech Emotion Recognition. Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414489 ↗
9. Peng, Z., Lu, Y., Pan, S., and Liu, Y. (2021, January 6–11). Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention. Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414286 ↗
10. Speech Classification to Recognize Emotion Using Artificial Neural Network
11. Tzirakis, P., Zhang, J., and Schuller, B.W. (2018, January 15–20). End-to-End Speech Emotion Recognition Using Deep Neural Networks. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462677 ↗
12. Sarma, M., Ghahremani, P., Povey, D., Goel, N.K., and Dehak, N. (2018, January 2–6). Emotion Identification from Raw Speech Signals Using DNNs. Proceedings of the Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India.10.21437/Interspeech.2018-1353 ↗
13. Yu, Y., and Kim, Y.J. (2020). Attention-LSTM-Attention Model for Speech Emotion Recognition and Analysis of IEMOCAP Database. Electronics, 9.10.3390/electronics9050713 ↗
14. Speech emotion recognition with deep convolutional neural networks10.1016/j.bspc.2020.101894 ↗
15. Muppidi, A., and Radfar, M. (2021, January 6–11). Speech Emotion Recognition Using Quaternion Convolutional Neural Networks. Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414248 ↗
16. Unsupervised feature selection and NMF de-noising for robust Speech Emotion Recognition10.1016/j.apacoust.2020.107645 ↗
17. Tronchin, L., Kob, M., and Guarnaccia, C. (2018). Spatial Information on Voice Generation from a Multi-Channel Electroglottograph. Appl. Sci., 8.10.3390/app8091560 ↗
18. Fant, G. (1971). Acoustic Theory of Speech Production, De Gruyter Mouton.10.1515/9783110873429 ↗
19. Robust glottal activity detection using the phase of an electroglottographic signal10.1016/j.bspc.2017.03.007 ↗
20. Text-Independent Phoneme Segmentation Combining EGG and Speech Data10.1109/TASLP.2016.2533865 ↗
21. Electroglottographic Parameterization of the Effects of Gender, Vowel and Phonatory Registers on Vocal Fold Vibratory Patterns: An Indian Perspective10.1007/s12070-010-0099-0 ↗
22. Evaluation of the Electroglottographic signal variability by amplitude-speed combined analysis10.1016/j.bspc.2016.10.003 ↗
23. Modal and Nonmodal Voice Quality Classification Using Acoustic and Electroglottographic Features10.1109/TASLP.2017.2759002 ↗
24. Comparison of parametrization methods of electroglottographic and inverse filtered acoustic speech pressure signals in distinguishing between phonation types10.1016/j.bspc.2017.04.001 ↗
25. The dynamics of vocal onset10.1016/j.bspc.2019.01.004 ↗
26. Filipa, M., and Ternstrm, S. (2020). Flow ball-assisted voice training: Immediate effects on vocal fold contacting. Biomed. Signal Process. Control, 62.10.1016/j.bspc.2020.102064 ↗
27. Chen, L., Ren, J., Chen, P., Mao, X., and Zhao, Q. (2022). Limited text speech synthesis with electroglottograph based on Bi-LSTM and modified Tacotron-2. Appl. Intell.10.1007/s10489-021-03075-x ↗
28. Use of Electroglottograph (EGG) to Find a Relationship between Pitch, Emotion and Personality10.1016/j.promfg.2015.07.236 ↗
29. Speech emotional features extraction based on electroglottograph10.1162/NECO_a_00523 ↗
30. Prasanna, S.R.M., and Govind, D. (2010, January 26–30). Analysis of excitation source information in emotional speech. Proceedings of the 11th Annual Conference of the International Speech Communication Association, Chiba, Japan.10.21437/Interspeech.2010-284 ↗
31. Significance of incorporating excitation source parameters for improved emotion recognition from speech and electroglottographic signals10.1007/s10772-017-9445-x ↗
32. Distilling the Knowledge in a Neural Network
33. Afouras, T., Chung, J.S., and Zisserman, A. (2020, January 4–8). ASR is All You Need: Cross-Modal Distillation for Lip Reading. Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054253 ↗
34. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
35. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2015, January 7–9). FitNets: Hints for Thin Deep Nets. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
36. Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., and Choi, J.Y. (November, January 27). A Comprehensive Overhaul of Feature Distillation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00201 ↗
37. Albanie, S., Nagrani, A., Vedaldi, A., and Zisserman, A. (2018, January 22–26). Emotion Recognition in Speech Using Cross-Modal Transfer in the Wild. Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Korea.10.1145/3240508.3240578 ↗
38. Li, R., Zhao, J., and Jin, Q. (September, January 30). Speech Emotion Recognition via Multi-Level Cross-Modal Distillation. Proceedings of the Interspeech 2021, Brno, Czech Republic.10.21437/Interspeech.2021-785 ↗
39. Speech emotion recognition using deep 1D & 2D CNN LSTM networks10.1016/j.bspc.2018.08.035 ↗
40. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
41. Prosody modification using instants of significant excitation10.1109/TSA.2005.858051 ↗
42. Chen, L., Mao, X., and Compare, A. (2013, January 11–13). A new method for speech synthesis combined with EGG. Proceedings of the National Conference on Man-Machine Speech Communication 2013, Lianyungang, China.
43. F0 contour approximation model for a one-stream tonal word recognition system10.1016/j.aeue.2016.02.006 ↗
44. Content Classification With Electroglottograph10.1088/1742-6596/1544/1/012191 ↗
45. An Approach of Fundamental Frequencies Smoothing for Chinese Tone Recognition
46. Multi-level knowledge distillation for low-resolution object detection and facial expression recognition10.1016/j.knosys.2022.108136 ↗
47. Wu, J., Hua, Y., Yang, S., Qin, H., and Qin, H. (2019). Speech Enhancement Using Generative Adversarial Network by Distilling Knowledge from Statistical Method. Appl. Sci., 9.10.3390/app9163396 ↗
48. Super-resolution guided knowledge distillation for low-resolution image classification10.1016/j.patrec.2022.02.006 ↗
49. Wang, J., Zhang, P., He, Q., Li, Y., and Hu, Y. (2021). Revisiting Label Smoothing Regularization with Knowledge Distillation. Appl. Sci., 11.10.3390/app11104699 ↗
50. On Information and Sufficiency10.1214/aoms/1177729694 ↗
51. Annotations and consistency detection for Chinese dual-mode emotional speech database
52. Kingma, D., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
53. Core Affect, Prototypical Emotional Episodes, and Other Things Called Emotion: Dissecting the Elephant10.1037/0022-3514.76.5.805 ↗
54. Viualizing data using t-SNE
55. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., and Weiss, B. (2005, January 4–8). A database of german emotional speech. Proceedings of the Interspeech 2005—Eurospeech, 9th European Conference on Speech Communication and Technology, Lisbon, Portugal.10.21437/Interspeech.2005-446 ↗