Confidence-Based Knowledge Distillation to Reduce Training Costs and Carbon Footprint for Low-Resource Neural Machine Translation
Source abstract
Abstract
The transformer-based deep learning approach represents the current state-of-the-art in machine translation (MT) research. Large-scale pretrained transformer models produce state-of-the-art performance across a wide range of MT tasks for many languages. However, such deep neural network (NN) models are often data-, compute-, space-, power-, and energy-hungry, typically requiring powerful GPUs or large-scale clusters to train and deploy. As a result, they are often regarded as “non-green” and “unsustainable” technologies. Distilling knowledge from large deep NN models (teachers) to smaller NN models (students) is a widely adopted sustainable development approach in MT as well as in broader areas of natural language processing (NLP), including speech, and image processing. However, distilling large pretrained models presents several challenges. First, increased training time and cost that scales with the volume of data used for training a student model. This could pose a challenge for translation service providers (TSPs), as they may have limited budgets for training. Moreover, CO2 emissions generated during model training are typically proportional to the amount of data used, contributing to environmental harm. Second, when querying teacher models, including encoder–decoder models such as NLLB, the translations they produce for low-resource languages may be noisy or of low quality. This can undermine sequence-level knowledge distillation (SKD), as student models may inherit and reinforce errors from inaccurate labels. In this study, the teacher model’s confidence estimation is employed to filter those instances from the distilled training data for which the teacher exhibits low confidence. We tested our methods on a low-resource Urdu-to-English translation task operating within a constrained training budget in an industrial translation setting. Our findings show that confidence estimation-based filtering can significantly reduce the cost and CO2 emissions associated with training a student model without drop in translation quality, making it a practical and environmentally sustainable solution for the TSPs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Handwritten Text Recognition Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computing, South East Technological University, R93 V960 Carlow, Ireland
Faculty of Business, Technological University Dublin, D02HW71 Dublin, Ireland
L3i, University of La Rochelle, 17000 La Rochelle, France
Funding
South East Technological University
References (46)
Show source-provided references
1. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
2. Kim, Y., and Rush, A.M. (2016, January 1–5). Sequence-Level Knowledge Distillation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1139 ↗
3. Wei, J., Sun, L., Leng, Y., Tan, X., Yu, B., and Guo, R. (2024). Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation. arXiv.10.24963/ijcai.2024/722 ↗
4. Moslem, Y. (2025). Efficient Speech Translation through Model Compression and Knowledge Distillation. arXiv.
5. Team, N., Costa-jussà, M.R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., and Licht, D. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv.
6. Waheed, A., Kadaoui, K., Raj, B., and Abdul-Mageed, M. (2025). uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes. arXiv.10.18653/v1/2025.naacl-long.296 ↗
7. Britz, D., Le, Q., and Pryzant, R. (2017, January 7–8). Effective Domain Mixing for Neural Machine Translation. Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark.10.18653/v1/W17-4712 ↗
8. Gu, J., Bradbury, J., Xiong, C., Li, V.O.K., and Socher, R. (2018). Non-Autoregressive Neural Machine Translation. arXiv.
9. Kasai, J., Pappas, N., Peng, H., Cross, J., and Smith, N.A. (2020). Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. arXiv.
10. Knowledge distillation: A survey10.1007/s11263-021-01453-z ↗
11. Wang, F., Yan, J., Meng, F., and Zhou, J. (2021, January 1–6). Selective Knowledge Distillation for Neural Machine Translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.10.18653/v1/2021.acl-long.504 ↗
12. Zhou, C., Neubig, G., and Gu, J. (2021). Understanding Knowledge Distillation in Non-autoregressive Machine Translation. arXiv.
13. Bucila, C., Caruana, R., and Niculescu-Mizil, A. (2006, January 20–23). Model compression. Proceedings of the Knowledge Discovery and Data Mining, Philadelphia, PA, USA.10.1145/1150402.1150464 ↗
14. Mirzadeh, S., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. (2020, January 7–12). Improved knowledge distillation via teacher assistant. Proceedings of the AAAI 2020—34th AAAI Conference on Artificial Intelligence, New York, NY, USA.
15. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
16. Kim, Y.J., Junczys-Dowmunt, M., Hassan, H., Fikri Aji, A., Heafield, K., Grundkiewicz, R., and Bogoychev, N. (2019, January 4). From Research to Production and Back: Ludicrously Fast Neural Machine Translation. Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, China.10.18653/v1/D19-5632 ↗
17. Yoo, K.M., Park, D., Kang, J., Lee, S.W., and Park, W. (2021, January 16–20). GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic.10.18653/v1/2021.findings-emnlp.192 ↗
18. Schick, T., and Schütze, H. (2021). Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference. arXiv.10.18653/v1/2021.eacl-main.20 ↗
19. Zhou, Y., Maharjan, S., and Liu, B. (2023). Scalable Prompt Generation for Semi-supervised Learning with Language Models. arXiv.10.18653/v1/2023.findings-eacl.58 ↗
20. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv.
21. Ho, N., Schmid, L., and Yun, S.Y. (2023). Large Language Models Are Reasoning Teachers. arXiv.10.18653/v1/2023.acl-long.830 ↗
22. He, N., Lai, H., Zhao, C., Cheng, Z., Pan, J., Qin, R., Lu, R., Lu, R., Zhang, Y., and Zhao, G. (2024). TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise. arXiv.
23. Shridhar, K., Stolfo, A., and Sachan, M. (2023, January 9–14). Distilling Reasoning Capabilities into Smaller Language Models. Proceedings of the Findings of the Association for Computational Linguistics: ACL, Toronto, ON, Canada.10.18653/v1/2023.findings-acl.441 ↗
24. Hsieh, C.Y., Li, C.L., Yeh, C.k., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.Y., and Pfister, T. (2023, January 9–14). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Proceedings of the Findings of the Association for Computational Linguistics: ACL, Toronto, ON, Canada.10.18653/v1/2023.findings-acl.507 ↗
25. Wang, P., Li, L., Chen, L., Song, F., Lin, B., Cao, Y., Liu, T., and Sui, Z. (2023). Making Large Language Models Better Reasoners with Alignment. arXiv.
26. Li, Y., Yuan, P., Feng, S., Pan, B., Sun, B., Wang, X., Wang, H., and Li, K. (2023). Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data. arXiv.10.1609/aaai.v38i17.29821 ↗
27. Chen, H., Wu, S., Quan, X., Wang, R., Yan, M., and Zhang, J. (2023). MCC-KD: Multi-CoT Consistent Knowledge Distillation. arXiv.10.18653/v1/2023.findings-emnlp.454 ↗
28. Liu, W., Li, G., Zhang, K., Du, B., Chen, Q., Hu, X., Xu, H., Chen, J., and Wu, J. (2024). Mind’s Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models. arXiv.10.18653/v1/2024.naacl-long.376 ↗
29. Currey, A., Mathur, P., and Dinu, G. (2020, January 16–20). Distilling Multiple Domains for Neural Machine Translation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.364 ↗
30. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA.10.3115/1073083.1073135 ↗
31. Yu, D., Backurs, A., Gopi, S., Inan, H., Kulkarni, J., Lin, Z., Xie, C., Zhang, H., and Zhang, W. (2023, January 10–16). Training Private and Efficient Language Models with Synthetic Data from LLMs. Proceedings of the Socially Responsible Language Modelling Research, New Orleans, LA, USA.
32. Jooste, W., Haque, R., and Way, A. (2022). Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient. Information, 13.10.3390/info13020088 ↗
33. Li, J., Nag, S., Liu, H., Tang, X., Sarwar, S., Cui, L., Gu, H., Wang, S., He, Q., and Tang, J. (2025). Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data. arXiv.10.18653/v1/2025.findings-naacl.142 ↗
34. Koneru, S., Liu, D., and Niehues, J. (2022). Cost-Effective Training in Low-Resource Neural Machine Translation. arXiv.
35. A survey on deep learning and its applications
36. Balakrishnan, R., Geetha, V., Kumar, M., and Leung, M.F. (2023). Reduction in Residential Electricity Bill and Carbon Dioxide Emission through Renewable Energy Integration Using an Adaptive Feed-Forward Neural Network System and MPPT Technique. Sustainability, 15.10.3390/su151914088 ↗
37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv.
38. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.10.1109/CVPR.2016.90 ↗
39. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer Normalization. arXiv.
40. Zafar, M., Castaldo, A., Nayak, P., Haque, R., and Way, A. (2024, January 15–16). The SETU-ADAPT Submissions to WMT 2024 Chat Translation Tasks. Proceedings of the Ninth Conference on Machine Translation, Miami, FL, USA.10.18653/v1/2024.wmt-1.104 ↗
41. Post, M. (2018). A call for clarity in reporting BLEU scores. arXiv.10.18653/v1/W18-6319 ↗
42. Popović, M. (2015, January 17–18). chrF: Character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal.10.18653/v1/W15-3049 ↗
43. Rei, R., Stewart, C., Farinha, A.C., and Lavie, A. (2020, January 16–18). COMET: A Neural Framework for MT Evaluation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.213 ↗
44. Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019). Quantifying the carbon emissions of machine learning. arXiv.
45. Koehn, P. (2004, January 12–16). Statistical Significance Tests for Machine Translation Evaluation. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.
46. Neubig, G., Dou, Z., Hu, J., Michel, P., Pruthi, D., Wang, X., and Wieting, J. (2019). Compare-mt: A Tool for Holistic Comparison of Language Generation Systems. arXiv.10.18653/v1/N19-4007 ↗