Advancing Spanish Speech Emotion Recognition: A Comprehensive Benchmark of Pre-Trained Models
Source abstract
Abstract
Feature extraction for speech emotion recognition (SER) has evolved from handcrafted techniques through deep learning methods to embeddings derived from pre-trained models (PTMs). This study presents the first comparative analysis focused on using PTMs for Spanish SER, evaluating six models—Whisper, Wav2Vec 2.0, WavLM, HuBERT, TRILLsson, and CLAP—across six emotional speech databases: EmoMatchSpanishDB, MESD, MEACorpus, EmoWisconsin, INTER1SP, and EmoFilm. We propose a robust framework combining layer-wise feature extraction with Leave-One-Speaker-Out validation to ensure interpretable model comparisons. Our method significantly outperforms existing state-of-the-art benchmarks, notably achieving scores on metrics such as F1 on EmoMatchSpanishDB (88.32%), INTER1SP (99.83%), and MEACorpus (92.53%). Layer-wise analyses reveal optimal emotional representation extraction at early layers in 24-layer models and middle layers in larger architectures. Additionally, TRILLsson exhibits remarkable generalization in speaker-independent evaluations, highlighting the necessity of strategic model selection, fine-tuning, and language-specific adaptations to maximize SER performance for Spanish.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Speech and Audio Processing · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Facultad de Instrumentacion Electronica, Universidad Veracruzana, Cto. Gonzalo Aguirre Beltran S/N, Xalapa 91000, Mexico
Instituto Tecnologico Superior de Poza Rica, Tecnologico Nacional de Mexico, Luis Donaldo Colosio Murrieta S/N, Arroyo del Maiz, Poza Rica 93230, Mexico
Electronics Department, National Institute for Astrophysics, Optics and Electronics, Sta. María Tonantzintla, Puebla 72840, Mexico
Funding
The Secretariat of Science, Humanities, Technology, and Innovation (SECIHTI)
References (91)
Show source-provided references
1. Geethu, V., Vrindha, M.K., Anurenjan, P.R., Deepak, S., and Sreeni, K.G. (2023, January 19–21). Speech Emotion Recognition, Datasets, Features and Models: A Review. Proceedings of the 2023 International Conference on Control, Communication and Computing (ICCC), Thiruvananthapuram, India.10.1109/ICCC57789.2023.10165567 ↗
2. Toward detecting emotions in spoken dialogs10.1109/TSA.2004.838534 ↗
3. A survey of speech emotion recognition in natural environment10.1016/j.dsp.2020.102951 ↗
4. Deep learning approaches for speech emotion recognition: State of the art and research challenges10.1007/s11042-020-09874-7 ↗
5. Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations10.1016/j.inffus.2023.102019 ↗
6. A critical review of state-of-the-art chatbot designs and applications10.1002/widm.1434 ↗
7. Two-layer fuzzy multiple random forest for speech emotion recognition in human-robot interaction10.1016/j.ins.2019.09.005 ↗
8. Liu, Z., Hu, B., Li, X., Liu, F., Wang, G., and Yang, J. (2017, January 16–18). Detecting Depression in Speech Under Different Speaking Styles and Emotional Valences. Proceedings of the Brain Informatics, Beijing, China.10.1007/978-3-319-70772-3_25 ↗
9. Acoustical properties of speech as indicators of depression and suicidal risk10.1109/10.846676 ↗
10. Li, H.C., Pan, T., Lee, M.H., and Chiu, H.W. (2021). Make Patient Consultation Warmer: A Clinical Application for Speech Emotion Recognition. Appl. Sci., 11.10.3390/app11114782 ↗
11. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends10.1145/3129340 ↗
12. ICARUS: Source generator based real-time recognition of speech in noisy stressful and Lombard effect environments10.1016/0167-6393(95)00007-B ↗
13. Cai, Y., Li, X., and Li, J. (2023). Emotion Recognition Using Different Sensors, Emotion Models, Methods and Datasets: A Comprehensive Review. Sensors, 23.10.3390/s23052455 ↗
14. Yu, J., Zhang, C., Song, Y., and Cai, W. (2021). ICE-GAN: Identity-aware and Capsule-Enhanced GAN with Graph-based Reasoning for Micro-Expression Recognition and Synthesis. arXiv.10.1109/IJCNN52387.2021.9533988 ↗
15. A systematic review on affective computing: Emotion models, databases, and recent advances10.1016/j.inffus.2022.03.009 ↗
16. Pan-Cultural Elements in Facial Displays of Emotion10.1126/science.164.3875.86 ↗
17. Darwin, C. (1998). The Expression Of The Emotions In Man And Animals, Oxford University Press.10.1093/oso/9780195112719.002.0002 ↗
18. Bălan, O., Moise, G., Petrescu, L., Moldoveanu, A., Leordeanu, M., and Moldoveanu, F. (2020). Emotion Classification Based on Biophysical Signals and Machine Learning Techniques. Symmetry, 12.10.3390/sym12010021 ↗
19. Zhou, E., Zhang, Y., and Duan, Z. (2024, January 14–19). Learning Arousal-Valence Representation from Categorical Emotion Labels of Speech. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10445890 ↗
20. A Comprehensive Review of Speech Emotion Recognition Systems10.1109/ACCESS.2021.3068045 ↗
21. Doğdu, C., Kessler, T., Schneider, D., Shadaydeh, M., and Schweinberger, S.R. (2022). A Comparison of Machine Learning Algorithms and Feature Sets for Automatic Vocal Emotion Recognition in Speech. Sensors, 22.10.3390/s22197561 ↗
22. Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, H.G., and Schuller, B. (2011, January 22–27). Deep neural networks for acoustic emotion recognition: Raising the benchmarks. Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic.10.1109/ICASSP.2011.5947651 ↗
23. Speech Emotion Recognition Based on SVM and ANN10.18178/ijmlc.2018.8.3.687 ↗
24. Bagged support vector machines for emotion recognition from speech10.1016/j.knosys.2019.104886 ↗
25. Jain, M., Narayan, S., Balaji, P., P, B.K., Bhowmick, A., R, K., and Muthu, R.K. (2020). Speech Emotion Recognition using Support Vector Machine. arXiv.
26. Speech emotion recognition using hidden Markov models10.1016/S0167-6393(03)00099-2 ↗
27. Nwe, T.L., Foo, S.W., and Silva, L.C.D. (2003, January 15–18). Detection of stress and emotion in speech using traditional and FFT based log energy features. Proceedings of the Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint, Singapore.
28. Mao, S., Tao, D., Zhang, G., Ching, P.C., and Lee, T. (2019, January 12–17). Revisiting Hidden Markov Models for Speech Emotion Recognition. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683172 ↗
29. Neiberg, D., Elenius, K., and Laskowski, K. (2009, January 6–10). Emotion recognition in spontaneous speech using GMMs. Proceedings of the Interspeech, Brighton, UK.
30. Recognition of Emotions from Speech using Excitation Source Features10.1016/j.proeng.2012.06.394 ↗
31. Bhakre, S.K., and Bang, A. (2016, January 21–24). Emotion recognition on the basis of audio signal using Naive Bayes classifier. Proceedings of the 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Jaipur, India.10.1109/ICACCI.2016.7732408 ↗
32. Khan, A., and Roy, U.K. (2017, January 22–24). Emotion recognition using prosodie and spectral features of speech and Naïve Bayes Classifier. Proceedings of the 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), Chennai, India.10.1109/WiSPNET.2017.8299916 ↗
33. Machine learning for human emotion recognition: A comprehensive review10.1007/s00521-024-09426-2 ↗
34. Badshah, A.M., Ahmad, J., Rahim, N., and Baik, S.W. (2017, January 13–15). Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network. Proceedings of the 2017 International Conference on Platform Technology and Service (PlatCon), Busan, Republic of Korea.10.1109/PlatCon.2017.7883728 ↗
35. A CNN-Assisted Enhanced Audio Signal Processing for Speech Emotion Recognition10.3390/s20010183 ↗
36. Latif, S., Rana, R., Khalifa, S., Jurdak, R., and Epps, J. (2020). Direct Modelling of Speech Emotion from Raw Speech. arXiv.10.21437/Interspeech.2019-3252 ↗
37. Speech Emotion Classification Using Attention-Based LSTM10.1109/TASLP.2019.2925934 ↗
38. Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM10.1109/ACCESS.2020.2990405 ↗
39. Lee, J., and Tashev, I.J. (2015, January 6–10). High-level feature representation using recurrent neural network for speech emotion recognition. Proceedings of the Interspeech, Dresden, Germany.10.21437/Interspeech.2015-336 ↗
40. Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural Networks10.1109/TMM.2014.2360798 ↗
41. Lim, W., Jang, D., and Lee, T. (2016, January 13–15). Speech emotion recognition using convolutional and Recurrent Neural Networks. Proceedings of the 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), Jeju, Republic of Korea.10.1109/APSIPA.2016.7820699 ↗
42. Zhao, Z., Zheng, Y., Zhang, Z., Wang, H., Zhao, Y., and Li, C. (2018, January 2–6). Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1477 ↗
43. Qin, C., Schlemper, J., Caballero, J., Price, A., Hajnal, J.V., and Rueckert, D. (2018). Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction. arXiv.10.1007/978-3-319-59050-9_51 ↗
44. Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv.
45. Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv.10.1109/TASLP.2021.3122291 ↗
46. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing10.1109/JSTSP.2022.3188113 ↗
47. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv.
48. Amiriparian, S., Packań, F., Gerczuk, M., and Schuller, B.W. (2024). ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets. arXiv.10.21437/Interspeech.2024-280 ↗
49. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019). wav2vec: Unsupervised Pre-training for Speech Recognition. arXiv.10.21437/Interspeech.2019-1873 ↗
50. Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap10.1109/TPAMI.2023.3263585 ↗
51. Osman, M., Kaplan, D.Z., and Nadeem, T. (2024). SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition. arXiv.10.21437/Interspeech.2024-2440 ↗
52. Phukan, O.C., Kashyap, G.S., Buduru, A.B., and Sharma, R. (2024, January 1–5). Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?. Proceedings of the Interspeech, Kos, Greece.10.21437/Interspeech.2024-2233 ↗
53. Triantafyllopoulos, A., Batliner, A., Rampp, S., Milling, M., and Schuller, B. (2024, January 1–5). INTERSPEECH 2009 Emotion Challenge Revisited: Benchmarking 15 Years of Progress in Speech Emotion Recognition. Proceedings of the Interspeech, Kos, Greece.10.21437/Interspeech.2024-97 ↗
54. Pepino, L., Riera, P., and Ferrer, L. (2021). Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings. arXiv.10.21437/Interspeech.2021-703 ↗
55. Chen, L.W., and Rudnicky, A. (2023, January 4–10). Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095036 ↗
56. Gao, Y., Chu, C., and Kawahara, T. (2023, January 20–24). Two-stage Finetuning of Wav2vec 2.0 for Speech Emotion Recognition with ASR and Gender Pretraining. Proceedings of the Interspeech, Dublin, Ireland.10.21437/Interspeech.2023-756 ↗
57. Jaiswal, M., and Provost, E.M. (2023). Best Practices for Noise-Based Augmentation to Improve the Performance of Deployable Speech-Based Emotion Recognition Systems. arXiv.
58. Schrüfer, O., Milling, M., Burkhardt, F., Eyben, F., and Schuller, B. (2024). Are you sure? Analysing Uncertainty Quantification Approaches for Real-world Speech Emotion Recognition. arXiv.10.21437/Interspeech.2024-977 ↗
59. In-depth investigation of speech emotion recognition studies from past to present—The importance of emotion recognition from speech signal for AI–
60. Sahu, G. (2019). Multimodal Speech Emotion Recognition and Ambiguity Resolution. arXiv.
61. A comprehensive survey on emotion recognition based on electroencephalograph (EEG) signals10.1007/s11042-023-14489-9 ↗
62. Kerkeni, L., Serrestou, Y., Mbarki, M., Raoof, K., and Mahjoub, M.A. (, January 16–18January). Speech Emotion Recognition: Methods and Cases Study. Proceedings of the 10th International Conference on Agents and Artificial Intelligence, ICAART 2018, Madeira, Portugal.10.5220/0006611601750182 ↗
63. Ortega-Beltrán, E., Cabacas-Maso, J., Benito-Altamirano, I., and Ventura, C. (2024). Better Spanish Emotion Recognition In-the-wild: Bringing Attention to Deep Spectrum Voice Analysis. arXiv.
64. EmoMatchSpanishDB: Study of speech emotion recognition machine learning models in a new Spanish elicited database10.1007/s11042-023-15959-w ↗
65. Begazo, R., Aguilera, A., Dongo, I., and Cardinale, Y. (2024). A Combined CNN Architecture for Speech Emotion Recognition. Sensors, 24.10.3390/s24175797 ↗
66. D’Mello, S., Graesser, A., Schuller, B., and Martin, J.C. (2011). EmoWisconsin: An Emotional Children Speech Database in Mexican Spanish. Affective Computing and Intelligent Interaction, Proceedings of the Fourth International Conference, ACII 2011, Memphis, TN, USA, 9–12 October 2011, Springer.10.1007/978-3-642-24571-8 ↗
67. Casals-Salvador, M., Costa, F., India, M., and Hernando, J. (2024). BSC-UPC at EmoSPeech-IberLEF2024: Attention Pooling for Emotion Recognition. arXiv.
68. Spanish MEACorpus 2023: A multimodal speech–text corpus for emotion analysis in Spanish from natural environments10.1016/j.csi.2024.103856 ↗
69. Paredes-Valverde, M.A., and del Pilar Salas-Zárate, M. (2024, January 24). Team ITST at EmoSPeech-IberLEF2024: Multimodal Speech-text Emotion Recognition in Spanish Forum. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024) at SEPLN 2024, Salamanca, Spain.
70. Esteban-Romero, S., Bellver-Soler, J., Martín-Fernández, I., Gil-Martín, M., D’Haro, L.F., and Fernández-Martínez, F. (2024, January 24). THAU-UPMatEmoSPeech-IberLEF2024: Efficient Adaptation of Mono-modal and Multi-modal Large Language Models for Automatic Speech Emotion Recognition. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024) at SEPLN 2024, Salamanca, Spain.
71. Cedeño-Moreno, D., Vargas-Lombardo, M., Delgado-Herrera, A., Caparrós-Láiz, C., and Bernal-Beltrán, T. (2024, January 24). UTPat EmoSPeech–IberLEF2024: Using Random Forest with FastText and Wav2Vec 2.0 for Emotion Detection. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024) at SEPLN 2024, Salamanca, Spain.
72. Duville, M.M., Alonso-Valerdi, L.M., and Ibarra-Zárate, D.I. (2021, January 1–5). The Mexican Emotional Speech Database (MESD): Elaboration and assessment based on machine learning. Proceedings of the 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Mexico City, Mexico.10.1109/EMBC46164.2021.9629934 ↗
73. Hozjan, V., Kacic, Z., Moreno, A., Bonafonte, A., and Nogueiras, A. (2002, January 29–31). Interface Databases: Design and Collection of a Multilingual Emotional Speech Database. Proceedings of the International Conference on Language Resources and Evaluation, Las Palmas, Spain.
74. Parada-Cabaleiro, E., Costantini, G., Batliner, A., Baird, A., and Schuller, B. (2018, January 2–6). Categorical vs Dimensional Perception of Italian Emotional Speech. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-47 ↗
75. Shor, J., and Venugopalan, S. (2022). TRILLsson: Distilled Universal Paralinguistic Speech Representations. arXiv.10.21437/Interspeech.2022-118 ↗
76. Wu, Y., Chen, K., Zhang, T., Hui, Y., Nezhurina, M., Berg-Kirkpatrick, T., and Dubnov, S. (2024). Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. arXiv.10.1109/ICASSP49357.2023.10095969 ↗
77. Evaluating Self-Supervised Speech Representations for Speech Emotion Recognition10.1109/ACCESS.2022.3225198 ↗
78. Macary, M., Tahon, M., Estève, Y., and Rousseau, A. (2020). On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition. arXiv.10.1109/SLT48900.2021.9383456 ↗
79. Hozjan, V., Kacic, Z., Moreno, A., Bonafonte, A., and Nogueiras, A. (2024, December 12). Emotional Speech Synthesis Database, ELRA catalogue. Available online: https://catalog.elra.info/en-us/repository/browse/ELRA-S0329/.
80. Chetia Phukan, O., Balaji Buduru, A., and Sharma, R. (2023, January 20–24). Transforming the Embeddings: A Lightweight Technique for Speech Emotion Recognition Tasks. Proceedings of the Interspeech, Dublin, Ireland.10.21437/Interspeech.2023-2561 ↗
81. Szeghalmy, S., and Fazekas, A. (2023). A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning. Sensors, 23.10.3390/s23042333 ↗
82. Scikit-learn: Machine Learning in Python
83. KNN Classification With One-Step Computation
84. Somvanshi, M., Chavan, P., Tambade, S., and Shinde, S.V. (2016, January 12–13). A review of machine learning techniques using decision tree and support vector machine. Proceedings of the 2016 International Conference on Computing Communication Control and automation (ICCUBEA), Pune, India.10.1109/ICCUBEA.2016.7860040 ↗
85. Samui, P., Sekhar, S., and Balas, V.E. (2017). Chapter 27—Support Vector Machine: Principles, Parameters, and Applications. Handbook of Neural Computation, Academic Press.
86. Du, K.L., Leung, C.S., Mow, W.H., and Swamy, M.N.S. (2022). Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era. Mathematics, 10.10.3390/math10244730 ↗
87. Pasad, A., Chou, J.C., and Livescu, K. (2021, January 13–17). Layer-Wise Analysis of a Self-Supervised Speech Representation Model. Proceedings of the 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Cartagena, Colombia.10.1109/ASRU51503.2021.9688093 ↗
88. Rethinking of BERT sentence embedding for text classification10.1007/s00521-024-10212-3 ↗
89. Botelho, C., Gimeno-Gómez, D., Teixeira, F., Mendonça, J., Pereira, P., Nunes, D.A.P., Rolland, T., Pompili, A., Solera-Ureña, R., and Ponte, M. (2024). Tackling Cognitive Impairment Detection from Speech: A submission to the PROCESS Challenge. arXiv.
90. Ng, S.I., Xu, L., Siegert, I., Cummins, N., Benway, N.R., Liss, J., and Berisha, V. (2024). A Tutorial on Clinical Speech AI Development: From Data Collection to Model Validation. arXiv.
91. Arai, K. (2024). Predictive Models in the Diagnosis of Parkinson’s Disease Through Voice Analysis. Proceedings of the Intelligent Systems and Applications, 2024, Springer.