Low-Resource Speech Recognition by Fine-Tuning Whisper with Optuna-LoRA
Source abstract
Abstract
In low-resource speech recognition, the performance of the Whisper model is often limited by the size of the available training data. To address this challenge, this paper proposes a training optimization method for the Whisper model that integrates Low-Rank Adaptation (LoRA) with the Optuna hyperparameter optimization framework. This combined approach enables efficient fine-tuning and enhances model performance. A dual-metric early stopping strategy, based on validation loss and relative word error rate improvement, is introduced to ensure robust convergence during training. Experimental data were collected from three low-resource languages in Xinjiang, China: Uyghur, Kazakh, and Kyrgyz. Compared to baseline LoRA fine-tuning, the proposed optimization method reduces WER by 20.98%, 6.46%, and 8.72%, respectively, across the three languages. The dual-metric early stopping strategy effectively shortens optimization time while preventing overfitting. Overall, these results demonstrate that the proposed method significantly reduces both WER and computational costs, making it highly effective for low-resource speech recognition tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Control Engineering, Southwest University of Science and Technology, Mianyang 621010, China
Funding
Mianyang Science Foundation · the National Natural Science Foundation of China
References (38)
Show source-provided references
1. Automatic speech recognition: Systematic literature review10.1109/ACCESS.2021.3112535 ↗
2. Yadav, H., and Sitaram, S. (2022). A survey of multilingual models for automatic speech recognition. arXiv.10.63317/4brvowuiwynn ↗
3. Unsupervised automatic speech recognition: A review10.1016/j.specom.2022.02.005 ↗
4. Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. (2020). Mls: A large-scale multilingual dataset for speech research. arXiv.10.21437/Interspeech.2020-2826 ↗
5. Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., Von Platen, P., Saraf, Y., and Pino, J. (2021). XLS-R: Self-supervised cross-lingual speech representation learning at scale. arXiv.10.21437/Interspeech.2022-143 ↗
6. Improving automatic speech recognition performance for low-resource languages with self-supervised models10.1109/JSTSP.2022.3184480 ↗
7. Slam, W., Li, Y., and Urouvas, N. (2023). Frontier research on low-resource speech recognition technology. Sensors, 23.10.3390/s23229096 ↗
8. DNN-based feature extraction technique for low-resource speech recognition
9. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network10.1016/j.physd.2019.132306 ↗
10. A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects10.1109/TNNLS.2021.3084827 ↗
11. End-to-end speech recognition: A survey10.1109/TASLP.2023.3328283 ↗
12. Huang, D., Yan, C., Li, Q., and Peng, X. (2024). From large language models to large multimodal models: A literature review. Appl. Sci., 14.10.3390/app14125068 ↗
13. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., and Dong, Z. (2023). A survey of large language models. arXiv.
14. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
15. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
16. Exploration of Whisper fine-tuning strategies for low-resource ASR10.1186/s13636-024-00349-3 ↗
17. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23–29). Robust speech recognition via large-scale weak supervision. Proceedings of the International Conference on Machine Learning, PMLR, Honolulu, HI, USA.
18. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S.Q. (2024). Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv.
19. Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., and Yang, F. (2024). Parameter-efficient fine-tuning in large models: A survey of methodologies. arXiv.10.21203/rs.3.rs-5393239/v1 ↗
20. Lora: Low-rank adaptation of large language models
21. Helora: Lora-heterogeneous federated fine-tuning for foundation models10.1145/3723877 ↗
22. Wang, X., Aitchison, L., and Rudolph, M. (2023). LoRA ensembles for large language model fine-tuning. arXiv.
23. Sicard, C., Pyszkowski, K., and Gillioz, V. (2023). Spaiche: Extending state-of-the-art ASR models to Swiss German dialects. arXiv.
24. Liu, W., Qin, Y., Peng, Z., and Lee, T. (2024, January 14–19). Sparsely shared lora on whisper for child speech recognition. Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447004 ↗
25. Polat, H., Turan, A.K., Koçak, C., and Ulaş, H.B. (2024). Implementation of a Whisper Architecture-Based Turkish Automatic Speech Recognition (ASR) System and Evaluation of the Effect of Fine-Tuning with a Low-Rank Adaptation (LoRA) Adapter on Its Performance. Electronics, 13.10.3390/electronics13214227 ↗
26. Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019, January 4–8). Optuna: A next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.10.1145/3292500.3330701 ↗
27. hyOPTXg: OPTUNA hyper-parameter optimization framework for predicting cardiovascular disease using XGBoost10.1016/j.bspc.2021.103456 ↗
28. Lai, L.H., Lin, Y.L., Liu, Y.H., Lai, J.P., Yang, W.C., Hou, H.P., and Pai, P.F. (2024). The use of machine learning models with Optuna in disease prediction. Electronics, 13.10.3390/electronics13234775 ↗
29. Early stopping by correlating online indicators in neural networks10.1016/j.neunet.2022.11.035 ↗
30. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
31. Biderman, D., Portes, J., Ortiz, J.J.G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., and Frankle, J. (2024). Lora learns less and forgets less. arXiv.
32. Shuttleworth, R., Andreas, J., Torralba, A., and Sharma, P. (2024). Lora vs full fine-tuning: An illusion of equivalence. arXiv.
33. Roze, A., Yin, S., Zhang, Z., Wang, D., and Hamdulla, A. (2015, January 1–11). THUGY20: A free Uyghur speech database. Proceedings of the NCMMSC’15, Tianjin, China.
34. Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F.M., and Weber, G. (2019). Common voice: A massively-multilingual speech corpus. arXiv.
35. Conneau, A., Ma, M., Khanuja, S., Zhang, Y., Axelrod, V., Dalmia, S., Riesa, J., Rivera, C., and Bapna, A. (2023, January 9–12). Fleurs: Few-shot learning evaluation of universal representations of speech. Proceedings of the 2022 IEEE Spoken Language Technology Workshop (SLT), Doha, Qatar.10.1109/SLT54892.2023.10023141 ↗
36. Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. (2025, December 01). PEFT: State-of-the-art Parameter-Efficient Fine-Tuning Methods. Available online: https://github.com/huggingface/peft.
37. Reimers, N., and Gurevych, I. (2018). Why comparing single performance scores does not allow to draw conclusions about machine learning approaches. arXiv.
38. Yeh, A. (2000). More accurate tests for the statistical significance of result differences. arXiv.10.3115/992730.992783 ↗