Speaker-Attributed Training for Multi-Speaker Speech Recognition Using Multi-Stage Encoders and Attention-Weighted Speaker Embedding
Source abstract
Abstract
Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic and Electrical Engineering, Kyungpook National University, Daegu 41566, Republic of Korea
Funding
Institute of Information & Communications Technology Planning & Evaluation (IITP)
References (45)
Show source-provided references
1. Some experiments on the recognition of speech, with one and with two ears10.1121/1.1907229 ↗
2. Hershey, J.R., Chen, Z., Roux, J.L., and Watanabe, S. (2016, January 20–25). Deep clustering: Discriminative embeddings for segmentation and separation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7471631 ↗
3. Yu, D., Kolbæk, M., Tan, Z.H., and Jensen, J. (2017, January 5–9). Permutation invariant training of deep models for speaker-independent multi-talker speech separation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952154 ↗
4. Chang, X., Qian, Y., and Yu, D. (2018, January 2–6). Monaural multi-talker speech recognition with attention mechanism and gated convolutional networks. Proceedings of the Interspeech, ISCA, Hyderabad, India.10.21437/Interspeech.2018-1547 ↗
5. Chang, X., Qian, Y., Yu, K., and Watanabe, S. (2019, January 12–17). End-to-end monaural multi-speaker ASR system without pretraining. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8682822 ↗
6. Chang, X., Zhang, W., Qian, Y., Roux, J.L., and Watanabe, S. (2020, January 4–8). End-to-end multi-speaker speech recognition with transformer. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054029 ↗
7. Denisov, P., and Vu, N.T. (2019, January 15–19). End-to-end multi-speaker speech recognition using speaker embeddings and transfer learning. Proceedings of the Interspeech, ISCA, Graz, Austria.10.21437/Interspeech.2019-1130 ↗
8. Kanda, N., Ye, G., Gaur, Y., Wang, X., Meng, Z., Chen, Z., and Yoshioka, T. (September, January 30). End-to-end speaker-attributed ASR with transformer. Proceedings of the Interspeech, ISCA, Brno, Czechia.
9. Kanda, N., Wu, J., Wu, Y., Xiao, X., Meng, Z., Wang, X., Gaur, Y., Chen, Z., Li, J., and Yoshioka, T. (2022, January 18–22). Streaming speaker-attributed ASR with token-level speaker embeddings. Proceedings of the Interspeech, ISCA, Incheon, Korea.10.21437/Interspeech.2022-253 ↗
10. Front-end factor analysis for speaker verification10.1109/TASL.2010.2064307 ↗
11. Wan, L., Wang, Q., Papir, A., and Moreno, I.L. (2018, January 15–20). Generalized end-to-end loss for speaker verification. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462665 ↗
12. Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., and Khudanpur, S. (2018, January 15–20). x-vectors: Robust DNN embeddings for speaker recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461375 ↗
13. Cosentino, J., Pariente, M., Cornell, S., Deleforge, A., and Vincent, E. (2020). LibriMix: An open-source dataset for generalizable speech separation. arXiv.
14. Zhu, Y., Ko, T., Snyder, D., Mak, B., and Povey, D. (2018, January 2–6). Self-attentive speaker embeddings for text-independent speaker verification. Proceedings of the Interspeech, ISCA, Hyderabad, India.10.21437/Interspeech.2018-1158 ↗
15. Park, T.J., Kanda, N., Dimitriadis, D., Han, K.J., Watanabe, S., and Narayanan, S. (2022). A review of speaker diarization: Recent advances with deep learning. Comput. Speech Lang., 72.10.1016/j.csl.2021.101317 ↗
16. Ning, Y., He, S., Wu, Z., Xing, C., and Zhang, L.J. (2019). A review of deep learning based speech synthesis. Appl. Sci., 9.10.3390/app9194050 ↗
17. Supervised speech separation based on deep learning: An overview10.1109/TASLP.2018.2842159 ↗
18. Distributed representations of words and phrases and their compositionality
19. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019, January 15–19). wav2vec: Unsupervised pre-training for speech recognition. Proceedings of the Interspeech, ISCA, Graz, Austria.10.21437/Interspeech.2019-1873 ↗
20. wav2vec 2.0: A framework for self-supervised learning of speech representations
21. Garcia-Romero, D., and Espy-Wilson, C.Y. (2011, January 27–31). Analysis of i-vector length normalization in speaker recognition systems. Proceedings of the Interspeech, ISCA, Florence, Italy.10.21437/Interspeech.2011-53 ↗
22. Ioffe, S. (2006, January 7–13). Probabilistic linear discriminant analysis. Proceedings of the European Conference on Computer Vision (ECCV), Graz, Austria.10.1007/11744085_41 ↗
23. Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks10.1109/TASLP.2017.2726762 ↗
24. Gu, Z., Liao, L., Chen, K., and Lu, J. (2020). Target speech extraction based on blind source separation and x-vector-based speaker selection trained with data augmentation. arXiv.
25. Chen, Z., Kanda, N., Wu, J., Wu, Y., Wang, X., Yoshioka, T., Li, J., Sivasankaran, S., and Eskimez, S.E. (2023, January 4–10). Speech separation with large-scale self-supervised learning. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096876 ↗
26. Meng, L., Kang, J., Cui, M., Wang, Y., Wu, X., and Meng, H. (2023, January 4–10). A sidecar separator can convert a single-talker speech recognition system to a multi-talker one. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095295 ↗
27. Deep neural networks for single-channel multi-talker speech recognition10.1109/TASLP.2015.2444659 ↗
28. Yu, D., Chang, X., and Qian, Y. (2017, January 20–24). Recognizing multi-talker speech with permutation invariant training. Proceedings of the Interspeech, ISCA, Stockholm, Sweden.10.21437/Interspeech.2017-305 ↗
29. Tan, T., Qian, Y., and Yu, D. (2018, January 15–20). Knowledge transfer in permutation invariant training for single-channel multi-talker speech recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461883 ↗
30. Huang, Z., Raj, D., García, P., and Khudanpur, S. (2023, January 4–10). Adapting self-supervised models to multi-talker speech recognition using speaker embeddings. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10097139 ↗
31. Single-channel multi-talker speech recognition with permutation invariant training10.1016/j.specom.2018.09.003 ↗
32. Kanda, N., Chang, X., Gaur, Y., Wang, X., Meng, Z., Chen, Z., and Yoshioka, T. (2021, January 19–22). Investigation of end-to-end speaker-attributed ASR for continuous multi-talker recordings. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383600 ↗
33. Variani, E., Lei, X., McDermott, E., Moreno, I.L., and Gonzalez-Dominguez, J. (2014, January 4–9). Deep neural networks for small footprint text-dependent speaker verification. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854363 ↗
34. Wang, Q., Muckenhirn, H., Wilson, K., Sridhar, P., Wu, Z., Hershey, J., Saurous, R.A., Weiss, R.J., Jia, Y., and Moreno, I.L. (2019, January 15–19). VoiceFilter: Targeted voice separation by speaker-conditioned spectrogram masking. Proceedings of the Interspeech, ISCA, Graz, Austria.10.21437/Interspeech.2019-1101 ↗
35. Cui, C., Sheikh, I., Sadeghi, M., and Vincent, E. (2023, January 16–20). End-to-end multichannel speaker-attributed ASR: Speaker guided decoder and input feature analysis. Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Taipei, Taiwan.10.1109/ASRU57964.2023.10389729 ↗
36. Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. (2015, January 19–24). Librispeech: An ASR corpus based on public domain audio books. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia.10.1109/ICASSP.2015.7178964 ↗
37. Wichern, G., Antognini, J., Flynn, M., Zhu, L.R., McQuinn, E., Crow, D., Manilow, E., and Roux, J.L. (2019, January 15–19). WHAM!: Extending speech separation to noisy environments. Proceedings of the Interspeech, ISCA, Graz, Austria.10.21437/Interspeech.2019-2821 ↗
38. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, K.N., Hannemann, M., Motlíček, P., Qian, Y., and Schwarz, P. (2011, January 11–15). The Kaldi speech recognition toolkit. Proceedings of the IEEE workshop on Automatic Speech Recognition and Understanding (ASRU), Waikoloa, Hawaii, USA.
39. Ravanelli, M., Parcollet, T., and Bengio, Y. (2019, January 12–17). The pytorch-Kaldi speech recognition toolkit. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683713 ↗
40. Nagrani, A., Chung, J.S., and Zisserman, A. (2017, January 20–24). VoxCeleb: A large-scale speaker identification dataset. Proceedings of the Interspeech, ISCA, Stockholm, Sweden.10.21437/Interspeech.2017-950 ↗
41. Pytorch: An imperative style, high-performance deep learning library
42. Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., Soplin, N.E.Y., Heymann, J., Wiesner, M., and Chen, N. (2018, January 2–6). ESPnet: End-to-end speech processing toolkit. Proceedings of the Interspeech, ISCA, Hyderabad, India.10.21437/Interspeech.2018-1456 ↗
43. Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Soplin, N.E.Y., Yamamoto, R., and Wang, X. (2019, January 14–18). A Comparative Study on Transformer vs RNN in Speech Applications. Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Singapore.10.1109/ASRU46091.2019.9003750 ↗
44. Karita, S., Soplin, N.E.Y., Watanabe, S., Delcroix, M., Ogawa, A., and Nakatani, T. (2019, January 15–19). Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration. Proceedings of the Interspeech, ISCA, Graz, Austria.10.21437/Interspeech.2019-1938 ↗
45. Li, S., Ouyang, B., Tong, F., Liao, D., Li, L., and Hong, Q. (September, January 30). Real-time end-to-end monaural multi-speaker speech recognition. Proceedings of the Interspeech, ISCA, Brno, Czechia.