Enhancing Far-Field Speech Recognition with Mixer: A Novel Data Augmentation Approach
Source abstract
Abstract
Recent advancements in end-to-end (E2E) modeling have notably improved automatic speech recognition (ASR) systems; however, far-field speech recognition (FSR) remains challenging due to signal degradation from factors such as low signal-to-noise ratio, reverberation, and interfering sounds. This requires richer training data and multi-channel speech enhancement. To address this gap, we introduce Mixer, a novel data augmentation technique designed to further enhance the performance of large-scale pre-trained models for FSR. Mixer interpolates and mixes feature representations of speech samples and their corresponding losses, extending the MixSpeech framework to intermediate layers of Whisper. Additionally, we propose Mixer-C, which further leverages multi-channel information by combining speech from different microphone channels using a channel selector. Experimental results demonstrate that Mixer significantly outperforms existing methods, including SpecAugment, achieving a relative word error rate (WER) reduction of 3.6% compared to the baseline. Furthermore, Mixer-C offers an additional WER improvement of 2.2%, showcasing its efficacy in improving FSR accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Speech Recognition and Synthesis · Advanced Data Compression Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Systems Engineering, Information Engineering University, Zhengzhou 450001, China
Laboratory for Advanced Computing and Intelligence Engineering, Wuxi 214000, China
Research and Development Department, Zhengzhou Xinda Institute of Advanced Technology, Zhengzhou 450001, China
Funding
National Natural Science Foundation of China · Natural Science Foundation of Henan Province · Henan Zhongyuan Science and Technology Innovation Leading Talent Project
References (34)
Show source-provided references
1. Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020, January 6–12). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Virtual.
2. Robust Speech Recognition via Large-Scale Weak Supervision
3. Pratap, V., Tjandra, A., Shi, B., Tomasello, P., Babu, A., Kundu, S., Elkahky, A.M., Ni, Z., Vyas, A., and Fazel-Zarandi, M. (2023). Scaling Speech Technology to 1000+ Languages. arXiv.
4. Saon, G., Tüske, Z., Audhkhasi, K., and Kingsbury, B. (2019, January 12–17). Sequence Noise Injected Training for End-to-end Speech Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, UK.10.1109/ICASSP.2019.8683706 ↗
5. Kubin, G., and Kacic, Z. (2019, January 15–19). SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. Proceedings of the 20th Annual Conference of the International Speech Communication Association, Interspeech 2019, Graz, Austria.
6. Schrank, T., Pfeifenberger, L., Zöhrer, M., Stahl, J., Mowlaee, P., and Pernkopf, F. (2016). Deep beamforming and data augmentation for robust speech recognition: Results of the 4th CHiME challenge. Proc. CHiME, 18–20.
7. Yalta, N., Watanabe, S., Hori, T., Nakadai, K., and Ogata, T. (2019, January 2–6). CNN-based multichannel end-to-end speech recognition for everyday home environments. Proceedings of the 2019 27th European Signal Processing Conference (EUSIPCO), A Coruna, Spain.10.23919/EUSIPCO.2019.8902524 ↗
8. Gaudesi, M., Weninger, F., Sharma, D., and Zhan, P. (2021, January 13–17). ChannelAugment: Improving Generalization of Multi-Channel ASR by Training with Input Channel Randomization. Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021, Cartagena, Colombia.10.1109/ASRU51503.2021.9687902 ↗
9. Meng, L., Xu, J., Tan, X., Wang, J., Qin, T., and Xu, B. (2021, January 6–11). MixSpeech: Data augmentation for low-resource automatic speech recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414483 ↗
10. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25–29). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event.
11. Karafiát, M., Veselỳ, K., Szöke, I., Mošner, L., Beneš, K., Witkowski, M., Barchi, G., and Pepino, L. (2023). BUT CHiME-7 system description. arXiv.10.21437/CHiME.2023-13 ↗
12. Kubin, G., and Kacic, Z. (2019, January 15–19). Examining the Combination of Multi-Band Processing and Channel Dropout for Robust Speech Recognition. Proceedings of the 20th Annual Conference of the International Speech Communication Association, Interspeech 2019, Graz, Austria.
13. Shimada, K., Takahashi, N., Takahashi, S., and Mitsufuji, Y. (2020). Sound event localization and detection using activity-coupled cartesian DOA vector and RD3Net. arXiv.
14. Kendall, A., and Gal, Y. (2017). What uncertainties do we need in bayesian deep learning for computer vision?. Adv. Neural Inf. Process. Syst., 5580–5590.
15. Zhang, H., Cissé, M., Dauphin, Y.N., and Lopez-Paz, D. (May, January 30). mixup: Beyond Empirical Risk Minimization, 2018. Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada.
16. Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y. (2019, January 9–15). Manifold mixup: Better representations by interpolating hidden states. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
17. Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., and Yoo, Y. (November, January 27). Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Republic of Korea.10.1109/ICCV.2019.00612 ↗
18. Faramarzi, M., Amini, M., Badrinaaraayanan, A., Verma, V., and Chandar, S. (March, January 22). PatchUp: A Feature-Space Block-Level Regularization Technique for Convolutional Neural Networks. Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022, Virtual Event.10.1609/aaai.v36i1.19938 ↗
19. Guo, H., Mao, Y., and Zhang, R. (2019). Augmenting Data with Mixup for Sentence Classification: An Empirical Study. arXiv.
20. Yoon, S., Kim, G., and Park, K. (2021, January 1–6). SSMix: Saliency-Based Span Mixup for Text Classification. Proceedings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event.10.18653/v1/2021.findings-acl.285 ↗
21. Chen, J., Yang, Z., and Yang, D. (2020, January 5–10). MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online.10.18653/v1/2020.acl-main.194 ↗
22. Chen, Y., Jiao, X., Zhang, H., Yang, X., and Qu, D. (2023, January 10–12). MixLoss: A Data Augmentation Method for Sequence Generation Tasks. Proceedings of the 2023 7th Asian Conference on Artificial Intelligence Technology (ACAIT), Jiaxing, China.10.1109/ACAIT60137.2023.10528597 ↗
23. Parameter-Efficient Transfer Learning for NLP
24. Thomas, B., Kessler, S., and Karout, S. (2022, January 23–27). Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746223 ↗
25. Yang, C.H., Li, B., Zhang, Y., Chen, N., Prabhavalkar, R., Sainath, T.N., and Strohman, T. (2023, January 4–10). From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece.10.1109/ICASSP49357.2023.10094903 ↗
26. Ko, H., and Hansen, J.H.L. (2022, January 18–22). WavPrompt: Towards Few-Shot Spoken Language Understanding with Frozen Language Models. Proceedings of the 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Republic of Korea.
27. Voice2Series: Reprogramming Acoustic Models for Time Series Classification
28. Carratino, L., Cissé, M., Jenatton, R., and Vert, J. (2020). On Mixup Regularization. arXiv.
29. Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., and Zou, J. How Does Mixup Help with Robustness and Generalization? In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, 3–7 May 2021.
30. Van Segbroeck, M., Zaid, A., Kutsenko, K., Huerta, C., Nguyen, T., Luo, X., Hoffmeister, B., Trmal, J., Omologo, M., and Maas, R. (2019). DiPCo–Dinner Party Corpus. arXiv.10.21437/Interspeech.2020-2800 ↗
31. Mu, B., Guo, P., Guo, D., Zhou, P., Chen, W., and Xie, L. (2023). Automatic channel selection and spatial feature integration for multi-channel speech recognition across various array topologies. arXiv.10.1109/ICASSP48485.2024.10447563 ↗
32. Drude, L., Heymann, J., Boeddeker, C., and Haeb-Umbach, R. (2018, January 10–12). NARA-WPE: A Python package for weighted prediction error dereverberation in Numpy and Tensorflow for online and offline processing. Proceedings of the Speech Communication; 13th ITG-Symposium, Oldenburg, Germany.
33. Acoustic beamforming for speaker diarization of meetings10.1109/TASL.2007.902460 ↗
34. Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., and Hajishirzi, H. (2018, January 8–14). Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01249-6_34 ↗