Noise Modeling to Build Training Sets for Robust Speech Enhancement
Source abstract
Abstract
DNN-based Speech Enhancement (SE) models suffer from significant performance degradation in real recordings due to the mismatch between the synthetic datasets employed for training and real test sets. To solve this problem, we propose a new Generative Adversarial Network framework for Noise Modeling (NM-GAN) that creates realistic paired training sets by imitating real noise distribution. The proposed framework combines a novel 7-layer U-Net with two bidirectional long short-term memory (LSTM) layers that act as a generator to construct complex noise. NM-GAN generates enough recall (diversity) and precision (noise quality) in its samples through adversarial and alternate training, effectively simulating real noise, which is then utilized to compose realistic paired training sets. Extensive experiments employing various qualitative and quantitative evaluation metrics verify the effectiveness of the generated noise samples and training sets, demonstrating our framework’s capabilities.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Music and Audio Processing · Acoustic Wave Phenomena Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing 100876, China
Key Laboratory of Computational Optical Imaging Technology, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100081, China
Funding
National Natural Science Foundation of China
References (28)
Show source-provided references
1. Loizou, P.C. (2017). Speech Enhancement: Theory and Practice, CRC Press.
2. Miao, Y., Gowayyed, M., and Metze, F. (2015, January 13–17). EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding. Proceedings of the 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Scottsdale, AZ, USA.10.1109/ASRU.2015.7404790 ↗
3. Sun, L., Du, J., Dai, L.R., and Lee, C.H. (2017, January 1–3). Multiple-target deep learning for LSTM-RNN based speech enhancement. Proceedings of the Hands-Free Speech Communications and Microphone Arrays (HSCMA), San Francisco, CA, USA.10.1109/HSCMA.2017.7895577 ↗
4. Generative Adversarial Networks
5. Pascual, S., Bonafonte, A., and Serrà, J. (2017, January 20–24). SEGAN: Speech Enhancement Generative Adversarial Network. Proceedings of the INTERSPEECH 2017, Stockholm, Sweden.10.21437/Interspeech.2017-1428 ↗
6. Jansson, A., Eric, J., and Humphrey, N. (2017, January 23–27). Singing voice separation with deep u-net convolutional networks. Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China.
7. Ernst, O., Chazan, S.E., Gannot, S., and Goldberger, J. (2018, January 3–7). Speech dereverberation using fully convolutional networks. Proceedings of the 2018 26th European Signal Processing Conference (EUSIPCO), Rome, Italy.10.23919/EUSIPCO.2018.8553141 ↗
8. Rethage, D., Pons, J., and Serra, X. (2018, January 22–27). A Wavenet for Speech Denoising. Proceedings of the ICASSP IEEE International Conference on Acoustics, Seoul, Korea.10.1109/ICASSP.2018.8462417 ↗
9. Reddy, C., Beyrami, E., Dubey, H., Gopal, V., Cheng, R., Cutler, R., Matusevych, S., Aichner, R., Aazami, A., and Braun, S. (2020). The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Speech Quality and Testing Framework. arXiv.10.21437/Interspeech.2020-3038 ↗
10. Chen, J., Chen, J., Chao, H., and Ming, Y. (2018, January 18–23). Image Blind Denoising with Generative Adversarial Network Based Noise Modeling. Proceedings of the IEEE/CVF Conference on Computer Vision & Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00333 ↗
11. GANs trained by a two time-scale update rule converge to a local Nash equilibrium
12. Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive growing of GANs for improved quality, stability, and variation. arXiv.
13. Are GANs created equal? A large-scale study
14. Improved techniques for training GANs
15. Shmelkov, K., Schmid, C., and Alahari, K. (2018, January 8–14). How good is my GAN?. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01216-8_14 ↗
16. Jaitly, N., and Hinton, G.E. (2013, January 16–21). Vocal tract length perturbation (VTLP) improves speech recognition. Proceedings of the 30th International Conference on Machine Learning (ICML), Atlanta, GA, USA.
17. ImageNet classification with deep convolutional neural networks
18. Cui, X., Goel, V., and Kingsbury, B. (2014, January 14). Data augmentation for deep convolutional neural network acoustic modeling. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, France.10.1109/ICASSP.2015.7178831 ↗
19. Mirza, M., and Osindero, S. (2014). Conditional Generative Adversarial Nets. Comput. Sci., 2672–2680.
20. Mao, X., Li, Q., Xie, H., Lau, R.Y.K., and Smolley, S.P. (2017, January 22–29). Least squares generative adversarial networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, France.10.1109/ICCV.2017.304 ↗
21. Baby, D., and Verhulst, S. (2019, January 12–17). Sergan: Speech Enhancement Using Relativistic Generative Adversarial Networks with Gradient Penalty. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683799 ↗
22. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
23. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27–30). Context encoders: Feature learning by inpainting. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.278 ↗
24. Burgy, L., Consel, C., Latry, F., Lawall, J.L., Palix, N., and Réveillère, L. (2000, January 5–9). Speech/non-speech classification using multiple features for robust endpoint detection. Proceedings of the 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing, Proceedings (Cat. No. 00CH37100), Washington, DC, USA.
25. Veaux, C., Yamagishi, J., and King, S. (2013, January 25–27). The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. Proceedings of the 2013 International Conference Oriental COCOSDA Held Jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA/CASLRE), Gurgaon, India.10.1109/ICSDA.2013.6709856 ↗
26. The diverse environments multi-channel acoustic noise database: A database of multichannel environmental noise recordings10.1121/1.4806631 ↗
27. Quackenbush, S.R., Barnwell, T.P., and Clements, M.A. (1988). Objective Measures of Speech Quality, Prentice-Hall.
28. Measuring speech quality for text-to-speech systems: Development and assessment of a modified mean opinion score (MOS) scale10.1016/j.csl.2003.12.001 ↗