Speech Emotion Recognition Using Transfer Learning: Integration of Advanced Speaker Embeddings and Image Recognition Models
Source abstract
Abstract
Automatic Speech Emotion Recognition (SER) plays a vital role in making human–computer interactions more natural and effective. A significant challenge in SER development is the limited availability of diverse emotional speech datasets, which hinders the application of advanced deep learning models. Transfer learning is a machine learning technique that helps address this issue by utilizing knowledge from pre-trained models to improve performance on a new task in a target domain, even with limited data. This study investigates the use of transfer learning from various pre-trained networks, including speaker embedding models such as d-vector, x-vector, and r-vector, and image classification models like AlexNet, GoogLeNet, SqueezeNet, ResNet-18, and ResNet-50. We also propose enhanced versions of the x-vector and r-vector models incorporating Multi-Head Attention Pooling and Angular Margin Softmax, alongside other architectural improvements. Additionally, reverberation from the Room Impulse Response datasets was added to the speech utterances to diversify and augment the available data. Notably, the enhanced r-vector model achieved classification accuracies of 74.05% Unweighted Accuracy (UA) and 73.68% Weighted Accuracy (WA) on the IEMOCAP dataset, and 80.25% UA and 79.81% WA on the CREMA-D dataset, outperforming the existing state-of-the-art methods. This study shows that using cross-domain transfer learning is beneficial for low-resource emotion recognition. The enhanced models developed in other domains (for non-emotional tasks) can further improve the accuracy of SER.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Speech Recognition and Synthesis · Emotion and Mood Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
University Science Park UNIZA, University of Žilina, Univerzitna 8215/1, 010 26 Žilina, Slovakia
Faculty of Electrical Engineering and Information Technology, University of Žilina, Univerzitna 8215/1, 010 26 Žilina, Slovakia
References (66)
Show source-provided references
1. Ekman, P. (2015). Emotion in the Human Face, Annotated ed., Malor Books.
2. SPHA: Smart Personal Health Advisor Based on Deep Analytics10.1109/MCOM.2018.1700274 ↗
3. Doctor, F., Karyotis, C., Iqbal, R., and James, A. (2016, January 6–9). An Intelligent Framework for Emotion Aware E-Healthcare Support Systems. Proceedings of the 2016 IEEE Symposium Series on Computational Intelligence (SSCI), Athens, Greece.10.1109/SSCI.2016.7850044 ↗
4. Tawari, A., and Trivedi, M. (2010, January 21–24). Speech Based Emotion Classification Framework for Driver Assistance System. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium, La Jolla, CA, USA.10.1109/IVS.2010.5547956 ↗
5. Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue10.1109/TVT.2004.830974 ↗
6. Customer Satisfaction Estimation in Contact Center Calls Based on a Hierarchical Multi-Task Model10.1109/TASLP.2020.2966857 ↗
7. Tzirakis, P., Zhang, J., and Schuller, B.W. (2018, January 15–20). End-to-End Speech Emotion Recognition Using Deep Neural Networks. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462677 ↗
8. Sarma, M., Ghahremani, P., Povey, D., Goel, N.K., Sarma, K.K., and Dehak, N. (2018, January 2–6). Emotion Identification from Raw Speech Signals Using DNNs. Proceedings of the Interspeech 2018, ISCA, Hyderabad, India.10.21437/Interspeech.2018-1353 ↗
9. Lieskovska, E., Jakubec, M., and Jarina, R. (2020, January 12–13). Speech Emotion Recognition Overview and Experimental Results. Proceedings of the 2020 18th International Conference on Emerging eLearning Technologies and Applications (ICETA), Kosice, Slovenia.10.1109/ICETA51985.2020.9379218 ↗
10. Yu, Y., and Kim, Y.-J. (2020). Attention-LSTM-Attention Model for Speech Emotion Recognition and Analysis of IEMOCAP Database. Electronics, 9.10.3390/electronics9050713 ↗
11. Autoencoder With Emotion Embedding for Speech Emotion Recognition10.1109/ACCESS.2021.3069818 ↗
12. Triantafyllopoulos, A., and Schuller, B.W. (2021, January 6–11). The Role of Task and Acoustic Similarity in Audio Transfer Learning: Insights from the Speech Emotion Recognition Case. Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2021), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414896 ↗
13. Snyder, D., Ghahremani, P., Povey, D., Garcia-Romero, D., Carmiel, Y., and Khudanpur, S. (2016, January 13–16). Deep Neural Network-Based Speaker Embeddings for End-to-End Speaker Verification. Proceedings of the 2016 IEEE Spoken Language Technology Workshop (SLT), San Diego, CA, USA.10.1109/SLT.2016.7846260 ↗
14. Li, C., Ma, X., Jiang, B., Li, X., Zhang, X., Liu, X., Cao, Y., Kannan, A., and Zhu, Z. (2017). Deep Speaker: An End-to-End Neural Speaker Embedding System. arXiv.
15. Zeinali, H., Wang, S., Silnova, A., Matějka, P., and Plchot, O. (2019). BUT System Description to VoxCeleb Speaker Recognition Challenge 2019. arXiv.
16. Pappagari, R., Wang, T., Villalba, J., Chen, N., and Dehak, N. (2020, January 4–8). X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition. Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Barcelona, Spain.10.1109/ICASSP40776.2020.9054317 ↗
17. Deep Speaker Embeddings for Speaker Verification: Review and Experimental Comparison10.1016/j.engappai.2023.107232 ↗
18. A Survey of Speech Emotion Recognition in Natural Environment10.1016/j.dsp.2020.102951 ↗
19. Dellaert, F., Polzin, T., and Waibel, A. (1996, January 3–6). Recognizing Emotion in Speech. Proceedings of the 4th International Conference on Spoken Language Processing (ICSLP ’96), Philadelphia, PA, USA.10.21437/ICSLP.1996-462 ↗
20. The Production and Recognition of Emotions in Speech: Features and Algorithms10.1016/S1071-5819(02)00141-6 ↗
21. Nwe, T.L., Foo, S.W., and De Silva, L.C. (2003, January 6–10). Classification of Stress in Speech Using Linear and Nonlinear Features. Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’03), Hong Kong, China.
22. Speech Emotion Recognition Based on SVM and ANN10.18178/ijmlc.2018.8.3.687 ↗
23. Fernandes, V., Mascarehnas, L., Mendonca, C., Johnson, A., and Mishra, R. (2018, January 23–24). Speech Emotion Recognition Using Mel Frequency Cepstral Coefficient and SVM Classifier. Proceedings of the 2018 International Conference on System Modeling & Advancement in Research Trends (SMART), Moradabad, India.10.1109/SYSMART.2018.8746939 ↗
24. Speech Communication Automatic Speech Emotion Recognition Using an Optimal Combination of Features Based on EMD-TKEO10.1016/j.specom.2019.09.002 ↗
25. Emotion Recognition in Speech Using Neural Networks10.1007/s005210070006 ↗
26. Lee, J., and Tashev, I. (2015, January 6–10). High-Level Feature Representation Using Recurrent Neural Network for Speech Emotion Recognition. Proceedings of the Interspeech 2015, ISCA, Dresden, Germany.10.21437/Interspeech.2015-336 ↗
27. Niu, Y., Zou, D., Niu, Y., He, Z., and Tan, H. (2018, January 12–14). Improvement on Speech Emotion Recognition Based on Deep Convolutional Neural Networks. Proceedings of the 2018 International Conference on Computing and Artificial Intelligence, Chengdu, China.10.1145/3194452.3194460 ↗
28. Satt, A., Rozenberg, S., and Hoory, R. (2017, January 20–24). Efficient Emotion Recognition from Speech Using Deep Learning on Spectrograms. Proceedings of the Interspeech 2017, ISCA, Stockholm, Sweden.10.21437/Interspeech.2017-200 ↗
29. A Systematic Survey on Multimodal Emotion Recognition Using Learning Algorithms
30. Antonaci, F.G., Olivetti, E.C., Marcolin, F., Castiblanco Jimenez, I.A., Eynard, B., Vezzetti, E., and Moos, S. (2024). Workplace Well-Being in Industry 5.0: A Worker-Centered Systematic Review. Sensors, 24.10.3390/s24175473 ↗
31. Deep Image Features in Music Information Retrieval10.2478/eletel-2014-0042 ↗
32. Amiriparian, S., Gerczuk, M., Ottl, S., Cummins, N., Freitag, M., Pugachevskiy, S., Baird, A., and Schuller, B. (2017, January 20–24). Snore Sound Classification Using Image-Based Deep Spectrum Features. Proceedings of the Interspeech 2017, ISCA, Stockholm, Sweden.10.21437/Interspeech.2017-434 ↗
33. Baird, A., Amiriparian, S., Cummins, N., Alcorn, A.M., Batliner, A., Pugachevskiy, S., Freitag, M., Gerczuk, M., and Schuller, B. (2017, January 20–24). Automatic Classification of Autistic Child Vocalisations: A Novel Database and Results. Proceedings of the Interspeech 2017, ISCA, Stockholm, Sweden.10.21437/Interspeech.2017-730 ↗
34. Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition10.1109/TCSVT.2017.2719043 ↗
35. Cummins, N., Amiriparian, S., Hagerer, G., Batliner, A., Steidl, S., and Schuller, B.W. (2017, January 23–27). An Image-Based Deep Spectrum Feature Representation for the Recognition of Emotional Speech. Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, CA, USA.10.1145/3123266.3123371 ↗
36. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
37. The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing10.1109/TAFFC.2015.2457417 ↗
38. Eyben, F., Weninger, F., Gross, F., and Schuller, B. (2013, January 21–25). Recent Developments in openSMILE, the Munich Open-Source Multimedia Feature Extractor. Proceedings of the 21st ACM International Conference on Multimedia, Barcelona, Spain. Available online: https://dl.acm.org/doi/10.1145/2502081.2502224.10.1145/2502081.2502224 ↗
39. Schmitt, M., Ringeval, F., and Schuller, B. (2016, January 8–12). At the Border of Acoustics and Linguistics: Bag-of-Audio-Words for the Recognition of Emotions in Speech. Proceedings of the Interspeech 2016, ISCA, San Francisco, CA, USA.10.21437/Interspeech.2016-1124 ↗
40. Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching10.1109/TMM.2017.2766843 ↗
41. Zhang, H., Gou, R., Shang, J., Shen, F., Wu, Y., and Dai, G. (2021). Pre-Trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition. Front. Physiol., 12.10.3389/fphys.2021.643202 ↗
42. Emotion-Aware Speaker Identification With Transfer Learning10.1109/ACCESS.2023.3297715 ↗
43. Xi, Y., Li, P., Song, Y., Jiang, Y., and Dai, L. (2019, January 18–21). Speaker to Emotion: Domain Adaptation for Speech Emotion Recognition with Residual Adapters. Proceedings of the 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Lanzhou, China.10.1109/APSIPAASC47483.2019.9023339 ↗
44. Padi, S., Sadjadi, S.O., Manocha, D., and Sriram, R.D. (2021, January 18–22). Improved Speech Emotion Recognition Using Transfer Learning and Spectrogram Augmentation. Proceedings of the 2021 International Conference on Multimodal Interaction (ICMI ‘21), Montréal, QC, Canada.10.1145/3462244.3481003 ↗
45. Feng, K., and Chaspari, T. (2020). A Review of Generalizable Transfer Learning in Automatic Emotion Recognition. Front. Comput. Sci., 2.10.3389/fcomp.2020.00009 ↗
46. Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. (2016). Progressive Neural Networks. arXiv.
47. Gideon, J., Khorram, S., Aldeneh, Z., Dimitriadis, D., and Provost, E.M. (2017). Progressive Neural Networks for Transfer Learning in Emotion Recognition. arXiv.10.21437/Interspeech.2017-1637 ↗
48. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019, January 9–15). Do ImageNet Classifiers Generalize to ImageNet?. Proceedings of the 36th International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
49. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2022). SqueezeNet: AlexNet-Level Accuracy with 50x Fewer Parameters and <0.5MB Model Size. arXiv.
50. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going Deeper with Convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
51. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
52. Variani, E., Lei, X., McDermott, E., Moreno, I.L., and Gonzalez-Dominguez, J. (2014, January 4–9). Deep Neural Networks for Small Footprint Text-Dependent Speaker Verification. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854363 ↗
53. Wang, Q., Downey, C., Wan, L., Mansfield, P.A., and Moreno, I.L. (2018, January 15–20). Speaker Diarization with LSTM. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462628 ↗
54. Peddinti, V., Povey, D., and Khudanpur, S. (2015, January 6–10). A Time Delay Neural Network Architecture for Efficient Modeling of Long Temporal Contexts. Proceedings of the Interspeech 2015, ISCA, Dresden, Germany.10.21437/Interspeech.2015-647 ↗
55. Povey, D., Cheng, G., Wang, Y., Li, K., Xu, H., Yarmohammadi, M., and Khudanpur, S. (2018, January 2–6). Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks. Proceedings of the Interspeech 2018, ISCA, Hyderabad, India.10.21437/Interspeech.2018-1417 ↗
56. Res2Net: A New Multi-Scale Backbone Architecture10.1109/TPAMI.2019.2938758 ↗
57. IEMOCAP: Interactive Emotional Dyadic Motion Capture Database10.1007/s10579-008-9076-6 ↗
58. CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset10.1109/TAFFC.2014.2336244 ↗
59. Element Analysis: A Wavelet-Based Method for Analysing Time-Localized Events in Noisy Time Series
60. Iandola, F.N., Ashraf, K., Moskewicz, M.W., and Keutzer, K. (2016, January 27–30). FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.284 ↗
61. Nagrani, A., Chung, J.S., and Zisserman, A. (2017, January 20–24). VoxCeleb: A Large-Scale Speaker Identification Dataset. Proceedings of the Interspeech 2017, ISCA, Stockholm, Sweden.10.21437/Interspeech.2017-950 ↗
62. Building and Evaluation of a Real Room Impulse Response Dataset10.1109/JSTSP.2019.2917582 ↗
63. Nöth, E., Horák, A., and Sojka, P. (2024). Enhancing Speech Emotion Recognition Using Transfer Learning from Speaker Embeddings. Text, Speech, and Dialogue, Springer Nature.10.1007/978-3-031-70563-2 ↗
64. Conversational Transfer Learning for Emotion Recognition10.1016/j.inffus.2020.06.005 ↗
65. Bouamor, H., Pino, J., and Bali, K. (2023). CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation. Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 6–10 December 2023, Association for Computational Linguistics.
66. Rogers, A., Boyd-Graber, J., and Okazaki, N. (2023). CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation. Findings of the Association for Computational Linguistics: ACL 2023, Toronto, ON, Canada, 9–14 July 2023, Association for Computational Linguistics.