Feature Integration Strategies for Neural Speaker Diarization in Conversational Telephone Speech
Source abstract
Abstract
This paper addresses the challenge of optimizing end-to-end neural diarization systems for conversational telephone speech, focusing on diverse acoustic features beyond traditional Mel-filterbanks. We present a methodological framework for integrating and analyzing different feature types as input to the well-known End-to-End Neural Diarization with Encoder Decoder Attractors (EEND-EDA) model, focusing on Emphasized Channel Attention, Propagation and Aggregation in Time Delay Neural Network (ECAPA-TDNN) embeddings and Geneva Minimalistic Acoustic Parameter Sets (GeMAPS). Our approach combines systematic feature analysis with adaptation strategies, including speaker-count restriction and regularization techniques. Moreover, through comprehensive ablation studies of GeMAPS features, we identify optimal acoustic parameters and temporal contexts for diarization tasks, achieving a reduced feature set that maintains performance while decreasing computational complexity. Experiments on the CallHome corpus demonstrate that our optimized ECAPA-TDNN with Mel-filterbank combination reduces Diarization Error Rate by 29% relative to baseline systems. Our evaluation framework extends beyond traditional metrics, revealing that different feature combinations exhibit distinct strengths in specific diarization aspects.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
AUDIAS, Electronic and Communication Technology Department, Escuela Politécnica Superior, Universidad Autónoma de Madrid, Av. Francisco Tomás y Valiente, 11, 28049 Madrid, Spain
Funding
Ministerio de Ciencia, Innovación y Universidades
References (44)
Show source-provided references
1. Raj, D., Denisov, P., Chen, Z., Erdogan, H., Huang, Z., He, M., Watanabe, S., Du, J., Yoshioka, T., and Luo, Y. (2021, January 19–22). Integration of Speech Separation, Diarization, and Recognition for Multi-Speaker Meetings: System Description, Comparison, and Analysis. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383556 ↗
2. O’Shaughnessy, D. (2025). Speaker Diarization: A Review of Objectives and Methods. Appl. Sci., 15.10.3390/app15042002 ↗
3. Gupta, A., and Purwar, A. (2022). Enhancing speaker diarization for audio-only systems using deep learning. Applications of Artificial Intelligence, Big Data and Internet of Things in Sustainable Development, CRC Press.10.1201/9781003245469-5 ↗
4. Menon, N.G., Shrivastava, A., Bhavana, N.D., and Simon, J. (2021, January 11–13). Deep Learning based Transcribing and Summarizing Clinical Conversations. Proceedings of the 2021 Fifth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Palladam, India.10.1109/I-SMAC52330.2021.9640683 ↗
5. O’Sullivan, J., Bogaarts, G., Schoenenberger, P., Tillmann, J., Slater, D., Mesgarani, N., Eule, E., Kilchenmann, T., Murtagh, L., and Hipp, J. (2023). Automatic speaker diarization for natural conversation analysis in autism clinical trials. Sci. Rep., 13.10.1038/s41598-023-36701-4 ↗
6. A review on speaker diarization systems and approaches10.1016/j.specom.2012.05.002 ↗
7. Kanda, N., Xiao, X., Gaur, Y., Wang, X., Meng, Z., Chen, Z., and Yoshioka, T. (2022, January 23–27). Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers Using End-to-End Speaker-Attributed ASR. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746225 ↗
8. Garcia-Romero, D., Snyder, D., Sell, G., Povey, D., and McCree, A. (2017, January 5–9). Speaker diarization using deep neural network embeddings. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953094 ↗
9. Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: Theory, implementation and analysis on standard tasks10.1016/j.csl.2021.101254 ↗
10. Chang, S.Y., Li, B., Simko, G., Sainath, T.N., Tripathi, A., van den Oord, A., and Vinyals, O. (2018, January 15–20). Temporal Modeling Using Dilated Convolution and Gating for Voice-Activity-Detection. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461921 ↗
11. Front-End Factor Analysis for Speaker Verification10.1109/TASL.2010.2064307 ↗
12. Dawalatabad, N., Ravanelli, M., Grondin, F., Thienpondt, J., Desplanques, B., and Na, H. (2021). ECAPA-TDNN Embeddings for Speaker Diarization. Proc. Interspeech, 3560–3564.10.21437/Interspeech.2021-941 ↗
13. Raj, D., Snyder, D., Povey, D., and Khudanpur, S. (2019, January 14–18). Probing the Information Encoded in X-Vectors. Proceedings of the 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Singapore.10.1109/ASRU46091.2019.9003979 ↗
14. Kwon, Y., Heo, H.S., Jung, J.W., Kim, Y.J., Lee, B.J., and Chung, J.S. (2022, January 23–27). Multi-Scale Speaker Embedding-Based Graph Attention Networks for Speaker Diarisation. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747450 ↗
15. A review of speaker diarization: Recent advances with deep learning10.1016/j.csl.2021.101317 ↗
16. An experimental review of speaker diarization methods with application to two-speaker conversational telephone speech recordings10.1016/j.csl.2023.101534 ↗
17. Fujita, Y., Kanda, N., Horiguchi, S., Nagamatsu, K., and Watanabe, S. (2019). End-to-End Neural Speaker Diarization with Permutation-Free Objectives. Proc. Interspeech, 4300–4304.10.21437/Interspeech.2019-2899 ↗
18. Fujita, Y., Kanda, N., Horiguchi, S., Xue, Y., Nagamatsu, K., and Watanabe, S. (2019, January 14–18). End-to-End Neural Speaker Diarization with Self-Attention. Proceedings of the 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Singapore.10.1109/ASRU46091.2019.9003959 ↗
19. Horiguchi, S., Fujita, Y., Watanabe, S., Xue, Y., and Nagamatsu, K. (2020). End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors. Proc. Interspeech, 269–273.10.21437/Interspeech.2020-1022 ↗
20. Xia, W., Lu, H., Wang, Q., Tripathi, A., Huang, Y., Moreno, I.L., and Sak, H. (2022, January 23–27). Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746531 ↗
21. Zhao, G., Wang, Q., Lu, H., Huang, Y., and Moreno, I.L. (2023, January 4–10). Augmenting Transformer-Transducer Based Speaker Change Detection with Token-Level Training Loss. Proceedings of the ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10094955 ↗
22. Kinoshita, K., Delcroix, M., and Tawara, N. (2021, January 6–11). Integrating End-to-End Neural and Clustering-Based Diarization: Getting the Best of Both Worlds. Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414333 ↗
23. Kinoshita, K., Delcroix, M., and Tawara, N. (September, January 30). Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. International Speech Communication Association, Brno, Czech Republic.
24. Combination of deep speaker embeddings for diarisation10.1016/j.neunet.2021.04.020 ↗
25. Medennikov, I., Korenevsky, M., Prisyach, T., Khokhlov, Y., Korenevskaya, M., Sorokin, I., Timofeeva, T., Mitrofanov, A., Andrusenko, A., and Podluzhny, I. (2020). Target-Speaker Voice Activity Detection: A Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario. Proc. Interspeech, 274–278.10.21437/Interspeech.2020-1602 ↗
26. Watanabe, S., Mandel, M., Barker, J., Vincent, E., Arora, A., Chang, X., Khudanpur, S., Manohar, V., Povey, D., and Raj, D. (2020, January 4). CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. Proceedings of the 6th International Workshop on Speech Processing in Everyday Environments (CHiME 2020), Online.10.21437/CHiME.2020-1 ↗
27. Wang, W., and Li, M. (2022, January 23–27). Incorporating End-to-End Framework Into Target-Speaker Voice Activity Detection. Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747772 ↗
28. Wang, D., Xiao, X., Kanda, N., Yoshioka, T., and Wu, J. (2023, January 4–10). Target Speaker Voice Activity Detection with Transformers and Its Integration with End-To-End Neural Diarization. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095185 ↗
29. Toyoshima, I., Okada, Y., Ishimaru, M., Uchiyama, R., and Tada, M. (2023). Multi-input speech emotion recognition model using mel spectrogram and GeMAPS. Sensors, 23.10.3390/s23031743 ↗
30. The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing10.1109/TAFFC.2015.2457417 ↗
31. Pepino, L., Riera, P., and Ferrer, L. (2021). Emotion Recognition from Speech Using wav2vec 2.0 Embeddings. Proc. Interspeech, 3400–3404.10.21437/Interspeech.2021-703 ↗
32. Atmaja, B.T., and Akagi, M. (2020, January 16–19). On the differences between song and speech emotion recognition: Effect of feature sets, feature types, and classifiers. Proceedings of the 2020 IEEE Region 10 Conference (TENCON), Osaka, Japan.10.1109/TENCON50793.2020.9293852 ↗
33. Alvarez-Trejos, J.I., Labrador, B., and Lozano-Diez, A. (2024, January 18–21). Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios. Proceedings of the Odyssey 2024: The Speaker and Language Recognition Workshop, Quebec, QC, Canada.10.21437/odyssey.2024-16 ↗
34. The NIST 1999 Speaker Recognition Evaluation—An Overview10.1006/dspr.1999.0355 ↗
35. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., and Schwarz, P. (2011, January 11–15). The Kaldi speech recognition toolkit. Proceedings of the IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. IEEE Signal Processing Society, Waikoloa, HI, USA.
36. Landini, F., Lozano-Diez, A., Diez, M., and Burget, L. (2022). From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization. Proc. Interspeech, 5095–5099.10.21437/Interspeech.2022-10451 ↗
37. Godfrey, J., Holliman, E., and McDaniel, J. (1992, January 23–26). SWITCHBOARD: Telephone speech corpus for research and development. Proceedings of the ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, San Francisco, CA, USA.10.1109/ICASSP.1992.225858 ↗
38. Przybocki, M., and Martin, A.F. (2004, January 1–3). NIST speaker recognition evaluation chronicles. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2004), Toledo, Spain.
39. Sadjadi, O., Greenberg, C., Singer, E., Mason, L., and Reynolds, D. (2024, April 25). NIST 2021 Speaker Recognition Evaluation Plan, NIST SRE, Available online: https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=932697.
40. Przybocki, M.A., Martin, A.F., and Le, A.N. (2006, January 28–30). NIST Speaker Recognition Evaluation Chronicles—Part 2. Proceedings of the 2006 IEEE Odyssey—The Speaker and Language Recognition Workshop, San Juan, PR, USA.10.1109/ODYSSEY.2006.248120 ↗
41. Martin, A.F., and Greenberg, C.S. (2009, January 6–10). NIST 2008 speaker recognition evaluation: Performance across telephone and room microphone channels. Proceedings of the Tenth Annual Conference of the International Speech Communication Association, Brighton, UK.10.21437/Interspeech.2009-679 ↗
42. Snyder, D., Chen, G., and Povey, D. (2015). Musan: A music, speech, and noise corpus. arXiv.
43. Ko, T., Peddinti, V., Povey, D., Seltzer, M.L., and Khudanpur, S. (2017, January 5–9). A study on data augmentation of reverberant speech for robust speech recognition. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953152 ↗
44. Bredin, H. (2023). Pyannote.audio 2.1 Speaker Diarization Pipeline: Principle, Benchmark, and Recipe. Proc. Interspeech, 1983–1987.10.21437/Interspeech.2023-105 ↗