Tongan Speech Recognition Based on Layer-Wise Fine-Tuning Transfer Learning and Lexicon Parameter Enhancement
Source abstract
Abstract
Speech recognition, as a key driver of artificial intelligence and global communication, has advanced rapidly in major languages, while studies on low-resource languages remain limited. Tongan, a representative Polynesian language, carries significant cultural value. However, Tongan speech recognition faces three main challenges: data scarcity, limited adaptability of transfer learning, and weak dictionary modeling. This study proposes improvements in adaptive transfer learning and NBPE-based dictionary modeling to address these issues. An adaptive transfer learning strategy with layer-wise unfreezing and dynamic learning rate adjustment is introduced, enabling effective adaptation of pretrained models to the target language while improving accuracy and efficiency. In addition, the MEA-AGA is developed by combining the Mind Evolutionary Algorithm (MEA) with the Adaptive Genetic Algorithm (AGA) to optimize the number of byte-pair encoding (NBPE) parameters, thereby enhancing recognition accuracy and speed. The collected Tongan speech data were expanded and preprocessed, after which the experiments were conducted on an NVIDIA RTX 4070 GPU (16 GB) using CUDA 11.8 under the Ubuntu 18.04 operating system. Experimental results show that the proposed method achieved a word error rate (WER) of 26.18% and a word-per-second (WPS) rate of 68, demonstrating clear advantages over baseline methods and confirming its effectiveness for low-resource language applications. Although the proposed approach demonstrates promising performance, this study is still limited by the relatively small corpus size and the early stage of research exploration. Future work will focus on expanding the dataset, refining adaptive transfer strategies, and enhancing cross-lingual generalization to further improve the robustness and scalability of the model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · ICT in Developing Communities · Phonetics and Phonology Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Beijing Research Institute of Automation for Machinery Industry Co., Ltd., No.1 Jiaochangkou Deshengmenwai, Xicheng District, Beijing 100120, China
School of Automation and Intelligence, Beijing Jiaotong University, No.3 Shangyuancun, Haidian District, Beijing 100044, China
Funding
The National Key R&D Program
References (58)
Show source-provided references
1. Deng, L., Hinton, G., and Kingsbury, B. (2013, January 26–31). New types of deep neural network learning for speech recognition and related applications: An overview. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, BC, Canada.10.1109/ICASSP.2013.6639344 ↗
2. Survey of Low-Resource Machine Translation10.1162/coli_a_00446 ↗
3. A survey on transfer learning10.1109/TKDE.2009.191 ↗
4. Graves, A., Mohamed, A., and Hinton, G. (2013, January 26–31). Speech recognition with deep recurrent neural networks. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, BC, Canada.10.1109/ICASSP.2013.6638947 ↗
5. Cho, J., Baskar, M.K., Li, R., Wiesner, M., Mallidi, S.H., and Yalta, N. (2018, January 18–21). Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.10.1109/SLT.2018.8639655 ↗
6. Singh, S., Hou, F., and Wang, R. (2023). A Novel Self-Training Approach for Low-Resource Speech Recognition. arXiv.10.21437/Interspeech.2023-540 ↗
7. Mukhamadiyev, A., Khujayarov, I., Djuraev, O., and Cho, J. (2022). Automatic Speech Recognition Method Based on Deep Learning Approaches for Uzbek Language. Sensors, 22.10.3390/s22103683 ↗
8. Feature Learning for Efficient ASR-Free Keyword Spotting in Low-Resource Languages10.1016/j.csl.2021.101275 ↗
9. Mamyrbayev, O., Oralbekova, D., Kydyrbekova, A., Turdalykyzy, T., and Bekarystankyzy, A. (2021, January 25–27). End-to-End Model Based on RNN-T for Kazakh Speech Recognition. Proceedings of the 2021 3rd International Conference on Computer Communication and the Internet (ICCCI), Nagoya, Japan.10.1109/ICCCI51764.2021.9486811 ↗
10. Zhi, T., Shi, Y., Du, W., Li, G., and Wang, D. (2021, January 18–20). M2ASR-MONGO: A Free Mongolian Speech Database and Accompanied Baselines. Proceedings of the 2021 24th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), Singapore.10.1109/O-COCOSDA202152914.2021.9660401 ↗
11. Oukas, N., Zerrouki, T., Haboussi, S., and Djettou, H. (2022, January 20–21). Arabic Speech Recognition Using Deep Learning and Common Voice Dataset. Proceedings of the 2022 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT), Sakheer, Bahrain.10.1109/3ICT56508.2022.9990834 ↗
12. End-to-End Speech Recognition of Tamil Language
13. Shetty, V.M., and Mysore Sathyendra, N.J. (2020, January 4–8). Improving the Performance of Transformer-Based Low-Resource Speech Recognition for Indian Languages. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053808 ↗
14. Pan, L., Li, S., Wang, L., and Dang, J. (2019, January 18–21). Effective Training of End-to-End ASR Systems for Low-Resource Lhasa Dialect of Tibetan Language. Proceedings of the 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Lanzhou, China.10.1109/APSIPAASC47483.2019.9023100 ↗
15. Anoop, C.S., and Ramakrishnan, A.G. (2021, January 27–30). CTC-Based End-to-End ASR for the Low-Resource Sanskrit Language with Spectrogram Augmentation. Proceedings of the 2021 National Conference on Communications (NCC), Bangalore, India.
16. Transfer Learning for Reinforcement Learning Domains: A Survey
17. Raghu, M., Zhang, C., Kleinberg, J., and Bengio, S. (2019). Transfusion: Understanding Transfer Learning for Medical Imaging. Advances in Neural Information Processing Systems 32, Neural Information Processing Systems Foundation, Inc.
18. Hussain, M., Bird, J.J., and Faria, D.R. (2018, January 5–7). A Study on CNN Transfer Learning for Image Classification. Proceedings of the 18th UK Workshop on Computational Intelligence (UKCI 2018), Nottingham, UK.10.1007/978-3-319-97982-3_16 ↗
19. A Perspective Survey on Deep Transfer Learning for Fault Diagnosis in Industrial Scenarios: Theories, Applications and Challenges10.1016/j.ymssp.2021.108487 ↗
20. Mainzinger, J., and Levow, G.-A. (2024, January 11–16). Fine-Tuning ASR Models for Very Low-Resource Languages: A Study on Mvskoke. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), Bangkok, Thailand.10.18653/v1/2024.acl-srw.16 ↗
21. Abdullah, A.A., Tabibian, S., Veisi, H., Mahmudi, A., and Rashid, T. (2024). End-to-End Transformer-Based Automatic Speech Recognition for Northern Kurdish: A Pioneering Approach. arXiv.
22. Cui, J., and Yang, J. (2023, January 17–19). Front-End Fusion and Large-Scale Weakly Supervised Decoding Module Based Myanmar Speech Recognition. Proceedings of the 2023 International Conference on Asian Language Processing (IALP), Singapore.10.1109/IALP61005.2023.10337260 ↗
23. Zeng, Z., Pham, V.T., Xu, H., Khassanov, Y., Chng, E.S., and Ni, C. (2021, January 24–27). Leveraging Text Data Using Hybrid Transformer-LSTM Based End-to-End ASR in Transfer Learning. Proceedings of the 2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP), Hong Kong, China.10.1109/ISCSLP49672.2021.9362086 ↗
24. Hjortnaes, N., Arkhangelskiy, T., Partanen, N., Rießler, M., and Tyers, F. (2020, January 11–12). Improving the Language Model for Low-Resource ASR with Online Text Corpora. Proceedings of the 1st Joint SLTU and CCURL Workshop (SLTU-CCURL 2020), Marseille, France.
25. Bansal, S., Kamper, H., Livescu, K., Lopez, A., and Goldwater, S. (2018). Pre-training on High-Resource Speech Recognition Improves Low-Resource Speech-to-Text Translation. arXiv.10.21437/Interspeech.2018-1326 ↗
26. Lee, H., Yoon, H.-W., Kim, J.-H., and Kim, J.-M. (2023). Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model. arXiv.10.21437/Interspeech.2023-934 ↗
27. Baller, T., Bennett, K., and Hamilton, H.J. (2021, January 25–28). Transfer Learning and Language Model Adaption for Low Resource Speech Recognition. Proceedings of the 34th Canadian Conference on Artificial Intelligence, Vancouver, BC, Canada.10.21428/594757db.d3394351 ↗
28. Meta-Learning for Indian Languages: Performance Analysis and Improvements with Linguistic Similarity Measures10.1109/ACCESS.2023.3300790 ↗
29. Yadav, H., and Sitaram, S. (2022, January 20–25). A Survey of Multilingual Models for Automatic Speech Recognition. Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC 2022), Paris, France.
30. Yu, Z., Zhang, Y., Qian, K., Fu, Y., and Lin, Y. (2023). Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning. arXiv.
31. Yan, B., Wiesner, M., Klejch, O., Jyothi, P., and Watanabe, S. (2022). Towards Zero-Shot Code-Switched Speech Recognition. arXiv.10.1109/ICASSP49357.2023.10097151 ↗
32. Li, X., Dalmia, S., Mortensen, D.R., Metze, F., and Black, A.W. (2019, January 6–9). Zero-Shot Learning for Speech Recognition with Universal Phonetic Model. Proceedings of the International Conference on Learning Representations (ICLR 2019), New Orleans, LA, USA.
33. Hsiao, R., Deng, L., McDermott, E., Travadi, R., and Zhuang, X. (2024, January 2–5). Optimizing Byte-Level Representation for End-to-End ASR. Proceedings of the 2024 IEEE Spoken Language Technology Workshop (SLT), Macao, China.10.1109/SLT61566.2024.10832272 ↗
34. Deng, L., Hsiao, R., and Ghoshal, A. (2022, January 23–27). Bilingual End-to-End ASR with Byte-Level Subwords. Proceedings of the ICASSP 2022—IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747842 ↗
35. End-to-End Speech Recognition Based on Thresholded BPE-Dropout Multi-Task Learning
36. Cai, Y., Wang, C., Ren, D., Zhu, Y., Zhang, J., and Nima, Z. (2024, January 25–28). End-to-End Tibetan Speech Recognition Study Based on Byte Pair Coding. Proceedings of the 23rd Chinese National Conference on Computational Linguistics (CCL 2024), Taiyuan, China.
37. Gaurav, M.S.A.I.S. (2021). Investigating Transformer Architecture for Automatic Speech Recognition and Machine Translation in Indian Languages. [Ph.D. Thesis, Indian Institute of Technology Madras].
38. Abu, T., Shi, Y., Zheng, T.F., and Wang, D. (2025, January 6–11). Sagalee: An Open Source Automatic Speech Recognition Dataset for Oromo Language. Proceedings of the 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India.10.1109/ICASSP49660.2025.10890761 ↗
39. Patel, T., and Scharenborg, O. (2022, January 18–22). Using Cross-Model Learnings for the Gram Vaani ASR Challenge 2022. Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH 2022), Incheon, Republic of Korea.10.21437/Interspeech.2022-10639 ↗
40. Abayomi-Alli, O.O., Damaševičius, R., Qazi, A., Adedoyin-Olowe, M., and Misra, S. (2022). Data Augmentation and Deep Learning Methods in Sound Classification: A Systematic Review. Electronics, 11.10.3390/electronics11223795 ↗
41. Enhanced Indonesian Ethnic Speaker Recognition Using Data Augmentation Deep Neural Network10.1016/j.jksuci.2021.04.002 ↗
42. Ramesh, V., Vatanparvar, K., Nemati, E., Nathan, V., Rahman, M.M., and Kuang, J. (2020, January 20–24). CoughGAN: Generating Synthetic Coughs That Improve Respiratory Disease Classification. Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Montréal, QC, Canada.10.1109/EMBC44109.2020.9175597 ↗
43. Unsupervised Feature Learning for Environmental Sound Classification Using Weighted Cycle-Consistent Generative Adversarial Network10.1016/j.asoc.2019.105912 ↗
44. Yella, N., and Rajan, B. (2021, January 22–25). Data Augmentation Using GAN for Sound-Based COVID-19 Diagnosis. Proceedings of the 2021 11th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), Cracow, Poland.10.1109/IDAACS53288.2021.9660990 ↗
45. Raj, S., Anant, A., Suryadevara, H., and Kumar, R.K. (2024, January 19–20). Hybrid Deep Learning Architecture with K-means Clustering for Weapon Detection in CCTV Surveillance. Proceedings of the 2024 IEEE International Conference on Computer Vision and Machine Intelligence (CVMI), Prayagraj, India.10.1109/CVMI61877.2024.10781588 ↗
46. Goel, A., Majumdar, A., Chouzenoux, E., and Chierchia, G. (2022, January 16–19). Deep Convolutional K-means Clustering. Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France.10.1109/ICIP46576.2022.9897742 ↗
47. Guo, W., Lin, K., and Ye, W. (2021, January 7–10). Deep Embedded K-means Clustering. Proceedings of the 2021 International Conference on Data Mining Workshops (ICDMW), Auckland, New Zealand.10.1109/ICDMW53433.2021.00090 ↗
48. Nam, Y., and Han, S. (2025). Random Forest Variable Importance-Based Selection Algorithm in Class Imbalance Problem. J. Classif., 1–14.10.1007/s00357-025-09512-7 ↗
49. Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization10.1016/j.knosys.2023.110851 ↗
50. Davari, M.R., Horoi, S., Natik, A., Lajoie, G., Wolf, G., and Belilovsky, E. (2022). Reliability of CKA as a Similarity Measure in Deep Learning. arXiv.
51. Stalker, P. (2010). A Guide to Countries of the World, Oxford University Press.
52. Moala, K. (2009). Search of the Friendly Islands, Pasifika Foundation Press.
53. Rhythm in the Kingdom: A Variationist Analysis of Speech Rhythm in Tongan English10.1017/S1360674323000540 ↗
54. Geng, J., Jia, D., He, Z., Wu, N., and Li, Z. (2024). Enhanced Conformer-Based Speech Recognition via Model Fusion and Adaptive Decoding with Dynamic Rescoring. Appl. Sci., 14.10.3390/app142411583 ↗
55. Zevallos, R., Bel, N., Cámbara, G., Farrús, M., and Luque, J. (2022). Data Augmentation for Low-Resource Quechua ASR Improvement. arXiv.10.21437/Interspeech.2022-770 ↗
56. Coto-Solano, R., Nicholas, S.A., Datta, S., Quint, V., Wills, P., Powell, E.N., Koka‘ua, L., Tanveer, S., and Feldman, I. (2022, January 20–25). Development of Automatic Speech Recognition for the Documentation of Cook Islands Māori. Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC 2022), Marseille, France.
57. Mak, F., Govender, A., and Badenhorst, J. (2024). Exploring ASR Fine-Tuning on Limited Domain-Specific Data for Low-Resource Languages. J. Digit. Humanit. Assoc. S. Afr., 5.10.55492/dhasa.v5i1.5024 ↗
58. Ren, Z., Yolwas, N., Slamu, W., Cao, R., and Wang, H. (2022). Improving Hybrid CTC/Attention Architecture for Agglutinative Language Speech Recognition. Sensors, 22.10.3390/s22197319 ↗