Target Speaker Localization Based on the Complex Watson Mixture Model and Time-Frequency Selection Neural Network
Source abstract
Abstract
Common sound source localization algorithms focus on localizing all the active sources in the environment. While the source identities are generally unknown, retrieving the location of a speaker of interest requires extra effort. This paper addresses the problem of localizing a speaker of interest from a novel perspective by first performing time-frequency selection before localization. The speaker of interest, namely the target speaker, is assumed to be sparsely active in the signal spectra. The target speaker-dominant time-frequency regions are separated by a speaker-aware Long Short-Term Memory (LSTM) neural network, and they are sufficient to determine the Direction of Arrival (DoA) of the target speaker. Speaker-awareness is achieved by utilizing a short target utterance to adapt the hidden layer outputs of the neural network. The instantaneous DoA estimator is based on the probabilistic complex Watson Mixture Model (cWMM), and a weighted maximum likelihood estimation of the model parameters is accordingly derived. Simulative experiments show that the proposed algorithm works well in various noisy conditions and remains robust when the signal-to-noise ratio is low and when a competing speaker exists.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Music and Audio Processing · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China
University of Chinese Academy of Sciences, Beijing 100190, China
Xinjiang Laboratory of Minority Speech and Language Information Processing, Xinjiang Technical Institute of Physics and Chemistry, Chinese Academy of Sciences, Urumchi 830001, China
Funding
National Natural Science Foundation of China · Strategic Priority Research Program of the Chinese Academy of Sciences · Key Science and Technology Project of the Xinjiang Uygur Autonomous Region · National 863 Program
References (37)
Show source-provided references
1. A survey on sound source localization in robotics: From binaural to array processing methods10.1016/j.csl.2015.03.003 ↗
2. Huang, Y., Benesty, J., and Elko, G.W. (2000, January 5–9). Passive acoustic source localization for video camera steering. Proceedings of the 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing, Istanbul, Turkey.
3. Design of robust superdirective arrays with a tunable tradeoff between directivity and frequency-invariance10.1109/TSP.2011.2106780 ↗
4. A binaural scene analyzer for joint localization and recognition of speakers in the presence of interfering noise sources and reverberation10.1109/TASL.2012.2193391 ↗
5. Busso, C., Hernanz, S., Chu, C.W., Kwon, S.i., Lee, S., Georgiou, P.G., Cohen, I., and Narayanan, S. (2005, January 23). Smart room: Participant and speaker localization and identification. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’05), Philadelphia, PA, USA.
6. The generalized correlation method for estimation of time delay10.1109/TASSP.1976.1162830 ↗
7. DiBiase, J.H., Silverman, H.F., and Brandstein, M.S. (2001). Robust localization in reverberant rooms. Microphone Arrays, Springer.10.1007/978-3-662-04619-7_8 ↗
8. Multiple emitter location and signal parameter estimation10.1109/TAP.1986.1143830 ↗
9. STFT Bin Selection for Localization Algorithms based on the Sparsity of Speech Signal Spectra
10. Araki, S., Nakatani, T., Sawada, H., and Makino, S. (2009, January 19–24). Blind sparse source separation for unknown number of sources using Gaussian mixture model fitting with Dirichlet prior. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan.10.1109/ICASSP.2009.4959513 ↗
11. Braun, S., Zhou, W., and Habets, E.A. (2015, January 18–21). Narrowband direction-of-arrival estimation for binaural hearing aids using relative transfer functions. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2015.7336917 ↗
12. Taseska, M., Lamani, G., and Habets, E.A. (2016). Online clustering of narrowband position estimates with application to multi-speaker detection and tracking. Advances in Machine Learning and Signal Processing, Springer.10.1007/978-3-319-32213-1_6 ↗
13. Blind separation of speech mixtures via time-frequency masking10.1109/TSP.2004.828896 ↗
14. Window-dominant signal subspace methods for multiple short-term speech source localization10.1109/TASLP.2016.2625458 ↗
15. Binaural localization of multiple sources in reverberant and noisy environments10.1109/TASL.2012.2183869 ↗
16. Wang, D., and Brown, G.J. (2006). Computational Auditory Scene Analysis: Principles, Algorithms, and Applications, Wiley-IEEE Press.
17. Pertilä, P., and Cakir, E. (2017, January 5–9). Robust direction estimation with convolutional neural networks based steered response power. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953333 ↗
18. Xu, C., Xiao, X., Sun, S., Rao, W., Chng, E.S., and Li, H. (2017, January 20–24). Weighted Spatial Covariance Matrix Estimation for MUSIC based TDOA Estimation of Speech Source. Proceedings of the Interspeech 2017, Stockholm, Sweden.10.21437/Interspeech.2017-199 ↗
19. Wang, Z.Q., Zhang, X., and Wang, D. (2018, January 2–6). Robust TDOA Estimation Based on Time-Frequency Masking and Deep Neural Networks. Proceedings of the Interspeech 2018, Hyderabad, India.10.21437/Interspeech.2018-1652 ↗
20. Supervised speech separation based on deep learning: An overview10.1109/TASLP.2018.2842159 ↗
21. Xiao, X., Zhao, S., Zhong, X., Jones, D.L., Chng, E.S., and Li, H. (2015, January 19–24). A learning-based approach to direction of arrival estimation in noisy and reverberant environments. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia.10.1109/ICASSP.2015.7178484 ↗
22. Takeda, R., and Komatani, K. (2016, January 13–16). Discriminative multiple sound source localization based on deep neural networks using independent location model. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), San Diego, CA, USA.10.1109/SLT.2016.7846325 ↗
23. Multiple-speaker localization based on direct-path features and likelihood maximization with spatial sparsity regularization10.1109/TASLP.2017.2740001 ↗
24. Sivasankaran, S., Vincent, E., and Fohr, D. (2018, January 2–6). Keyword-based speaker localization: Localizing a target speaker in a multi-speaker environment. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1526 ↗
25. Vu, D.H.T., and Haeb-Umbach, R. (2010, January 14–19). Blind speech separation employing directional statistics in an expectation maximization framework. Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Dallas, TX, USA.
26. Ito, N., Araki, S., Delcroix, M., and Nakatani, T. (2017, January 5–9). Probabilistic spatial dictionary based online adaptive beamforming for meeting recognition in noisy and reverberant environments. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952242 ↗
27. Drude, L., Jacob, F., and Haeb-Umbach, R. (September, January 31). DOA-estimation based on a complex watson kernel method. Proceedings of the 23rd European Signal Processing Conference (EUSIPCO), Nice, France.
28. Drude, L., Chinaev, A., Vu, D.H.T., and Haeb-Umbach, R. (2014, January 4–9). Source counting in speech mixtures using a variational EM approach for complex Watson mixture models. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854924 ↗
29. The complex Watson distribution and shape analysis10.1111/1467-9868.00210 ↗
30. Ito, N., Araki, S., and Nakatani, T. (2013, January 26–31). Permutation-free convolutive blind source separation via full-band clustering based on frequency-independent source presence priors. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6638256 ↗
31. Learning hidden unit contributions for unsupervised acoustic model adaptation10.1109/TASLP.2016.2560534 ↗
32. Žmolíková, K., Delcroix, M., Kinoshita, K., Higuchi, T., Ogawa, A., and Nakatani, T. (2017, January 16–20). Learning speaker representation for neural network based multichannel speaker extraction. Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Okinawa, Japan.10.1109/ASRU.2017.8268910 ↗
33. Ito, N., Araki, S., and Nakatani, T. (September, January 28). Data-driven and physical model-based designs of probabilistic spatial dictionary for online meeting diarization and adaptive beamforming. Proceedings of the European Signal Processing Conference (EUSIPCO), Kos, Greece.
34. Paul, D.B., and Baker, J.M. (1992, January 23–26). The design for the Wall Street Journal-based CSR corpus. Proceedings of the workshop on Speech and Natural Language, Harriman, NY, USA.10.3115/1075527.1075614 ↗
35. Wang, Z.Q., Le Roux, J., and Hershey, J.R. (2018, January 15–20). Multi-Channel Deep Clustering: Discriminative Spectral and Spatial Embeddings for Speaker-Independent Speech Separation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8461639 ↗
36. Image method for efficiently simulating small-room acoustics10.1121/1.382599 ↗
37. Veselỳ, K., Watanabe, S., Žmolíková, K., Karafiát, M., Burget, L., and Černockỳ, J.H. (2016, January 20–25). Sequence summarizing neural network for speaker adaptation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472692 ↗