Speech Enhancement Based on Unidirectional Interactive Noise Modeling Assistance
Source abstract
Abstract
It has been demonstrated that interactive speech and noise modeling outperforms traditional speech modeling-only methods for speech enhancement (SE). With a dual-branch topology that simultaneously predicts target speech and noise signals and employs bidirectional information communication between the two branches, the quality of the enhanced speech is significantly improved. However, the dual-branch topology greatly increases the model complexity and deployment cost, thus limiting its practicality. In this paper, we propose UniInterNet, a unidirectional information interaction-based dual-branch network to achieve noise modeling-assisted SE without any increase in complexity. Specifically, the noise branch still receives information from the speech branch to achieve more accurate noise modeling. Subsequently, the noise modeling results are utilized to assist the learning of the speech branch during backpropagation, while the speech branch no longer receives the auxiliary information from the noise branch, so only the speech branch is required during model deployment. Experimental results demonstrate that under the causal inference condition, the performance of UniInterNet only marginally decreases compared to the corresponding bidirectional information interaction scheme, while the model inference complexity is reduced by about 75%. With comparable overall performance, UniInterNet also outperforms previous interactive speech and noise modeling-based benchmarks in terms of causal inference and model complexity. Furthermore, UniInterNet surpasses other existing competitive methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Music and Audio Processing · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Xiaohongshu Inc., Shanghai 200020, China
Funding
Shenzhen Science and Technology Innovation Commission
References (49)
Show source-provided references
1. Défossez, A., Synnaeve, G., and Adi, Y. (2020, January 25–29). Real Time Speech Enhancement in the Waveform Domain. Proceedings of the Interspeech, Shanghai, China.10.21437/Interspeech.2020-2409 ↗
2. Pandey, A., and Wang, D. (2020, January 4–8). Densely Connected Neural Network with Dilated Convolutions for Real-Time Speech Enhancement in the Time Domain. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain.10.1109/ICASSP40776.2020.9054536 ↗
3. Kong, Z., Ping, W., Dantrey, A., and Catanzaro, B. (2022, January 22–27). Speech Denoising in the Waveform Domain With Self-Attention. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore.10.1109/ICASSP43922.2022.9746169 ↗
4. Tan, K., and Wang, D. (2018, January 2–6). A Convolutional Recurrent Neural Network for Real-Time Speech Enhancement. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1405 ↗
5. Choi, H.S., Kim, J., Huh, J., Kim, A., Ha, J.W., and Lee, K. (2019, January 6–9). Phase-Aware Speech Enhancement with Deep Complex U-Net. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
6. Fu, S., Liao, C., Tsao, Y., and Lin, S. (2019, January 18–20). MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement. Proceedings of the International Conference on Machine Learning and Machine Intelligence, Jakarta, Indonesia.
7. Hu, Y., Liu, Y., Lv, S., Xing, M., Zhang, S., Fu, Y., Wu, J., Zhang, B., and Xie, L. (2020, January 25–29). DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement. Proceedings of the Interspeech, Shanghai, China.10.21437/Interspeech.2020-2537 ↗
8. Two Heads are Better Than One: A Two-Stage Complex Spectral Mapping Approach for Monaural Speech Enhancement10.1109/TASLP.2021.3079813 ↗
9. Yan, X., Yang, Y., Guo, Z., Peng, L., and Xie, L. (2023, January 4–10). The NPU-Elevoc Personalized Speech Enhancement System for Icassp2023 DNS Challenge. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096362 ↗
10. Lu, Y.X., Ai, Y., and Ling, Z.H. (2023, January 20–24). MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra. Proceedings of the Interspeech, Dublin, Ireland.10.21437/Interspeech.2023-1441 ↗
11. Park, S.R., and Lee, J.W. (2017, January 20–24). A Fully Convolutional Neural Network for Speech Enhancement. Proceedings of the Interspeech, Stockholm, Sweden.10.21437/Interspeech.2017-1465 ↗
12. Gao, T., Du, J., Dai, L.R., and Lee, C.H. (2018, January 15–20). Densely Connected Progressive Learning for LSTM-Based Speech Enhancement. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461861 ↗
13. Cao, R., Abdulatif, S., and Yang, B. (2022, January 18–22). CMGAN: Conformer-based Metric GAN for Speech Enhancement. Proceedings of the Proceedings Interspeech, Incheon, Republic of Korea.10.36227/techrxiv.21187846.v2 ↗
14. DBT-Net: Dual-Branch Federative Magnitude and Phase Estimation with Attention-in-Attention Transformer for Monaural Speech Enhancement10.1109/TASLP.2022.3195112 ↗
15. Learning Complex Spectral Mapping with Gated Convolutional Recurrent Networks for Monaural Speech Enhancement10.1109/TASLP.2019.2955276 ↗
16. Le, X., Chen, H., Chen, K., and Lu, J. (September, January 30). DPCRN: Dual-Path Convolution Recurrent Network for Single Channel Speech Enhancement. Proceedings of the Interspeech, Brno, Czechia.
17. Zhao, S., Ma, B., Watcharasupat, K.N., and Gan, W.S. (2022, January 22–27). FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747578 ↗
18. Liu, J., and Zhang, X. (2023, January 4–10). ICCRN: Inplace Cepstral Convolutional Recurrent Neural Network for Monaural Speech Enhancement. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096918 ↗
19. Zhao, C., He, S., and Zhang, X. (2024, January 14–19). SICRN: Advancing Speech Enhancement through State Space Model and Inplace Convolution Techniques. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10446396 ↗
20. Li, A., Zheng, C., Zhang, L., and Li, X. (2022). Glance and gaze: A collaborative learning framework for single-channel speech enhancement. Appl. Acoust., 187.10.1016/j.apacoust.2021.108499 ↗
21. CompNet: Complementary network for single-channel speech enhancement10.1016/j.neunet.2023.09.041 ↗
22. Zhang, Y., Zou, H., and Zhu, J. (2024, January 14–19). A Two-Stage Framework in Cross-Spectrum Domain for Real-Time Speech Enhancement. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447096 ↗
23. Hao, X., Su, X., Horaud, R., and Li, X. (2021, January 6–11). Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414177 ↗
24. Chen, J., Wang, Z., Tuo, D., Wu, Z., Kang, S., and Meng, H. (2022, January 22–27). FullSubNet+: Channel Attention Fullsubnet with Complex Spectrograms for Speech Enhancement. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore.10.1109/ICASSP43922.2022.9747888 ↗
25. Chen, Z., and Zhang, P. (2022, January 18–22). Lightweight Full-band and Sub-band Fusion Network for Real Time Speech Enhancement. Proceedings of the Proceedings Interspeech, Incheon, Republic of Korea.10.21437/Interspeech.2022-153 ↗
26. Chen, J., Rao, W., Wang, Z., Lin, J., Wu, Z., Wang, Y., Shang, S., and Meng, H. (2023, January 4–10). Inter-Subnet: Speech Enhancement with Subband Interaction. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece.10.1109/ICASSP49357.2023.10094858 ↗
27. Hu, Y., Chen, C., Li, R., Zhu, Q., and Chng, E.S. (2024, January 1–5). Noise-aware Speech Enhancement using Diffusion Probabilistic Model. Proceedings of the Interspeech, Kos, Greece.10.21437/Interspeech.2024-929 ↗
28. Odelowo, B.O., and Anderson, D.V. (2018, January 15–20). A Study of Training Targets for Deep Neural Network-Based Speech Enhancement Using Noise Prediction. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8462126 ↗
29. Interactive Speech and Noise Modeling for Speech Enhancement
30. Two-Stage Learning and Fusion Network with Noise Aware for Time-Domain Monaural Speech Enhancement10.1109/LSP.2021.3105925 ↗
31. A Novel Noise-Aware Deep Learning Model for Underwater Acoustic Denoising10.1109/TGRS.2023.3336053 ↗
32. Discrete Cosine Transform10.1109/T-C.1974.223784 ↗
33. Valentini-Botinhao, C., Wang, X., Takaki, S., and Yamagishi, J. (2016, January 13–15). Investigating RNN-based speech enhancement methods for noise-robust Text-to-Speech. Proceedings of the ISCA Workshop on Speech Synthesis Workshop, Sunnyvale, CA, USA.10.21437/SSW.2016-24 ↗
34. Reddy, C.K., Gopal, V., Cutler, R., Beyrami, E., Cheng, R., Dubey, H., Matusevych, S., Aichner, R., Aazami, A., and Braun, S. (2020, January 25–29). The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results. Proceedings of the Proceedings Interspeech, Shanghai, China.10.21437/Interspeech.2020-3038 ↗
35. Binary and ratio time-frequency masks for robust speech recognition10.1016/j.specom.2006.09.003 ↗
36. Narayanan, A., and Wang, D. (2013, January 26–31). Ideal ratio mask estimation using deep neural networks for robust speech recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639038 ↗
37. Geng, C., and Wang, L. (2020, January 27–29). End-to-End Speech Enhancement Based on Discrete Cosine Transform. Proceedings of the 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), Dalian, China.10.1109/ICAICA50127.2020.9182513 ↗
38. Luo, Y., Chen, Z., and Yoshioka, T. (2020, January 4–8). Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054266 ↗
39. Veaux, C., Yamagishi, J., and King, S. (2013, January 25–27). The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. Proceedings of the 2013 International Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA/CASLRE), Gurgaon, India.10.1109/ICSDA.2013.6709856 ↗
40. Thiemann, J., Ito, N., and Vincent, E. (2013). The Diverse Environments Multi-channel Acoustic Noise Database (DEMAND): A database of multichannel environmental noise recordings. Proc. Meet. Acoust., 19.10.1121/1.4799597 ↗
41. Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., and Ritter, M. (2017, January 5–9). Audio Set: An ontology and human-labeled dataset for audio events. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952261 ↗
42. Pirker, G., Wohlmayr, M., Petrik, S., and Pernkopf, F. (2011, January 27–31). A pitch tracking corpus with evaluation on multipitch tracking scenario. Proceedings of the Interspeech, Florence, Italy.10.21437/Interspeech.2011-317 ↗
43. Recommendation, I. (2007). Wideband Extension to Recommendation p. 862 for the Assessment of Wideband Telephone Networks and Speech Codecs, International Telecommunication Union. Technical Report Standard 862.2.
44. Evaluation of Objective Quality Measures for Speech Enhancement10.1109/TASL.2007.911054 ↗
45. Rix, A., Beerends, J., Hollier, M., and Hekstra, A. (2001, January 7–11). Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Salt Lake City, UT, USA.
46. An Algorithm for Intelligibility Prediction of Time–Frequency Weighted Noisy Speech10.1109/TASL.2011.2114881 ↗
47. Roux, J.L., Wisdom, S., Erdogan, H., and Hershey, J.R. (2019, January 12–17). SDR—Half-baked or Well Done?. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8683855 ↗
48. Chen, J., Rao, W., Wang, Z., Wu, Z., Wang, Y., Yu, T., Shang, S., and Meng, H. (2022, January 18–22). Speech Enhancement with Fullband-Subband Cross-Attention Network. Proceedings of the Interspeech, Incheon, Republic of Korea.10.21437/Interspeech.2022-10257 ↗
49. Li, Y., Jin, X., Tong, L., Zhang, L.M., Yao, Y.Q., and Yan, H. (2024). A speech enhancement model based on noise component decomposition: Inspired by human cognitive behavior. Appl. Acoust., 221.10.1016/j.apacoust.2024.109997 ↗