Iteratively Refined Multi-Channel Speech Separation
Source abstract
Abstract
The combination of neural networks and beamforming has proven very effective in multi-channel speech separation, but its performance faces a challenge in complex environments. In this paper, an iteratively refined multi-channel speech separation method is proposed to meet this challenge. The proposed method is composed of initial separation and iterative separation. In the initial separation, a time–frequency domain dual-path recurrent neural network (TFDPRNN), minimum variance distortionless response (MVDR) beamformer, and post-separation are cascaded to obtain the first additional input in the iterative separation process. In iterative separation, the MVDR beamformer and post-separation are iteratively used, where the output of the MVDR beamformer is used as an additional input to the post-separation network and the final output comes from the post-separation module. This iteration of the beamformer and post-separation is fully employed for promoting their optimization, which ultimately improves the overall performance. Experiments on the spatialized version of the WSJ0-2mix corpus showed that our proposed method achieved a signal-to-distortion ratio (SDR) improvement of 24.17 dB, which was significantly better than the current popular methods. In addition, the method also achieved an SDR of 20.2 dB on joint separation and dereverberation tasks. These results indicate our method’s effectiveness and significance in the multi-channel speech separation field.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Advanced Adaptive Filtering Techniques · Blind Source Separation Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Speech and Audio Information Processing, School of Information Science and Technology, Beijing University of Technology, Beijing 100124, China
Funding
National Natural Science Foundation of China
References (29)
Show source-provided references
1. Chen, Z., Li, J., Xiao, X., Yoshioka, T., Wang, H., Wang, Z., and Gong, Y. (2017, January 16–20). Cracking the Cocktail Party Problem by Multi-Beam Deep Attractor Network. Proceedings of the 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Okinawa, Japan.10.1109/ASRU.2017.8268969 ↗
2. Past Review, Current Progress, and Challenges Ahead on the Cocktail Party Problem10.1631/FITEE.1700814 ↗
3. Chen, J., Mao, Q., and Liu, D. (2020). Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation. arXiv.10.21437/Interspeech.2020-2205 ↗
4. Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation10.1109/TASLP.2019.2915167 ↗
5. Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., and Zhong, J. (2021, January 6). Attention Is All You Need in Speech Separation. Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413901 ↗
6. Zhao, S., Ma, Y., Ni, C., Zhang, C., Wang, H., Nguyen, T.H., Zhou, K., Yip, J., Ng, D., and Ma, B. (2023). MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separation. arXiv.10.1109/ICASSP48485.2024.10445985 ↗
7. A Consolidated Perspective on Multimicrophone Speech Enhancement and Source Separation10.1109/TASLP.2016.2647702 ↗
8. Acoustic Beamforming for Speaker Diarization of Meetings10.1109/TASL.2007.902460 ↗
9. Ochiai, T., Delcroix, M., Ikeshita, R., Kinoshita, K., Nakatani, T., and Araki, S. (2020, January 4–8). Beam-TasNet: Time-Domain Audio Separation Network Meets Frequency-Domain Beamformer. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053575 ↗
10. Zhang, X., Wang, Z.-Q., and Wang, D. (2017, January 5–9). A Speech Enhancement Algorithm by Iterating Single- and Multi-Microphone Processing and Its Application to Robust ASR. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952161 ↗
11. Erdogan, H., Hershey, J.R., Watanabe, S., Mandel, M.I., and Roux, J.L. (2016, January 8). Improved MVDR Beamforming Using Single-Channel Mask Prediction Networks. Proceedings of the Interspeech 2016, ISCA, San Francisco, CA, USA.10.21437/Interspeech.2016-552 ↗
12. Towards Unified All-Neural Beamforming for Time and Frequency Domain Speech Separation10.1109/TASLP.2022.3229261 ↗
13. Xiao, X., Zhao, S., Jones, D.L., Chng, E.S., and Li, H. (2017, January 5–9). On Time-Frequency Mask Estimation for MVDR Beamforming with Application in Robust Speech Recognition. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952756 ↗
14. A Time-Domain Real-Valued Generalized Wiener Filter for Multi-Channel Neural Separation Systems10.1109/TASLP.2022.3205750 ↗
15. Zhang, X., Bao, C., Zhou, J., and Yang, X. (2023, January 14). A Beam-TFDPRNN Based Speech Separation Method in Reverberant Environments. Proceedings of the 2023 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), Zhengzhou, China.10.1109/ICSPCC59353.2023.10400232 ↗
16. Kavalerov, I., Wisdom, S., Erdogan, H., Patton, B., Wilson, K., Roux, J.L., and Hershey, J.R. (2019). Universal Sound Separation. arXiv.10.1109/WASPAA.2019.8937253 ↗
17. Tzinis, E., Wisdom, S., Hershey, J.R., Jansen, A., and Ellis, D.P.W. (2020, January 4–8). Improving Universal Sound Separation Using Sound Classification. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053921 ↗
18. Shi, Z., Liu, R., and Han, J. (2020). LaFurca: Iterative Refined Speech Separation Based on Context-Aware Dual-Path Parallel Bi-LSTM. arXiv.
19. Chen, H., Yi, Y., Feng, D., and Zhang, P. (2022). Beam-Guided TasNet: An Iterative Speech Separation Framework with Multi-Channel Output. arXiv.10.21437/Interspeech.2022-230 ↗
20. Wang, Z.-Q., Erdogan, H., Wisdom, S., Wilson, K., Raj, D., Watanabe, S., Chen, Z., and Hershey, J.R. (2021, January 19). Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383522 ↗
21. Yang, L., Liu, W., and Wang, W. (2022, January 23–27). TFPSNet: Time-Frequency Domain Path Scanning Network for Speech Separation. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747554 ↗
22. Yang, X., Bao, C., Zhang, X., and Chen, X. (2023, January 20). Monaural Speech Separation Method Based on Recurrent Attention with Parallel Branches. Proceedings of the INTERSPEECH 2023, ISCA, Dublin, Ireland.10.21437/Interspeech.2023-518 ↗
23. Wang, Z.-Q., Le Roux, J., and Hershey, J.R. (2018, January 15–20). Multi-Channel Deep Clustering: Discriminative Spectral and Spatial Embeddings for Speaker-Independent Speech Separation. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461639 ↗
24. Image Method for Efficiently Simulating Small-Room Acoustics10.1121/1.382599 ↗
25. Févotte, C., Gribonval, R., and Vincent, E. (2011). BSS_EVAL Toolbox User Guide Revision 2.0, IRISA.
26. Luo, Y., Ceolini, E., Han, C., Liu, S.-C., and Mesgarani, N. (2019). FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing. arXiv.10.1109/ASRU46091.2019.9003849 ↗
27. Quan, C., and Li, X. (2022, January 23). Multi-Channel Narrow-Band Deep Speech Separation with Full-Band Permutation Invariant Training. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747346 ↗
28. Chen, Z., Yoshioka, T., Lu, L., Zhou, T., Meng, Z., Luo, Y., Wu, J., Xiao, X., and Li, J. (2020, January 4–8). Continuous Speech Separation: Dataset and Analysis. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053426 ↗
29. Maciejewski, M., Wichern, G., McQuinn, E., and Roux, J.L. (2020, January 4–8). WHAMR!: Noisy and Reverberant Single-Channel Speech Separation. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053327 ↗