Exploring Selective Layer Freezing Strategies in Transformer Fine-Tuning: NLI Classifiers with Sub-3B Parameter Models
Source abstract
Abstract
In recent years, methods that selectively fine-tune or reduce the number of layers in large language models (LLMs) have garnered attention as an efficient alternative to traditional fine-tuning, where all layers are trained. In this study, we revisit the classical concept of layer freezing and propose a simple, effective strategy that selectively fine-tunes only a portion of transformer layers. We show that freezing the bottom 25% or 50% of layers in small-scale LLMs with sub-3 billion parameters yields significant improvements in memory efficiency and training speed while maintaining, or even surpassing, the performance of full fine-tuning and Low-Rank Adaptation (LoRA). Through experiments on Natural Language Inference (NLI) tasks using LLMs with fewer than 3 billion parameters, our approach achieves up to 50% memory savings and 30% faster training. Notably, our method does not require architectural modifications or additional parameters, making it particularly suitable for resource-constrained environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Magnetic Properties and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science and Engineering, Chungnam National University, Daejeon 34134, Republic of Korea
EurekaAI, Daejeon 34134, Republic of Korea
Funding
Institute of Information & Communications Technology Planning & Evaluation · Artificial Intelligence Convergence Innovation Human Resources Development · Chungnam National University
References (28)
Show source-provided references
1. Language Models are Few-Shot Learners
2. Palm: Scaling language modeling with pathways
3. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
4. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
5. Xie, S.M., Raghunathan, A., Liang, P., and Ma, T. (2022, January 25). An Explanation of In-context Learning as Implicit Bayesian Inference. Proceedings of the International Conference on Learning Representations, Virtual.
6. Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (2020). Pretrained Transformers Improve Out-of-Distribution Robustness. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 5–10 July 2020, Association for Computational Linguistics.
7. Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (2020). Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 5–10 July 2020, Association for Computational Linguistics.
8. Burstein, J., Doran, C., and Solorio, T. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2–7 June 2019.
9. Muresan, S., Nakov, P., and Villavicencio, A. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, 22–27 May 2022.
10. Tang, H., Chen, J., Zhang, W., and Guo, Z. (2024). Training Acceleration Method Based on Parameter Freezing. Electronics, 13.10.3390/electronics13112140 ↗
11. Bowman, S.R., Angeli, G., Potts, C., and Manning, C.D. (2015, January 17–21). A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1075 ↗
12. Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25–29). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the International Conference on Learning Representations, Virtual.
13. Dagan, I., Glickman, O., and Magnini, B. (2003, January 11–13). The pascal recognising textual entailment challenge. Proceedings of the Machine Learning Challenges Workshop, Southampton, UK.
14. Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., and Van Durme, B. (2018, January 5–6). Hypothesis Only Baselines in Natural Language Inference. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, New Orleans, LA, USA.10.18653/v1/S18-2023 ↗
15. Linzen, T., Chrupala, G., and Alishahi, A. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, 31 October–4 November 2018, Association for Computational Linguistics.
16. Howard, J., and Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv.10.18653/v1/P18-1031 ↗
17. Augenstein, I., Gella, S., Ruder, S., Kann, K., Can, B., Welbl, J., Conneau, A., Ren, X., and Rei, M. (2019). To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), Florence, Italy, 2 August 2019, Association for Computational Linguistics.
18. Gpt3.int8 (): 8-bit Matrix Multiplication for Transformers at Scale
19. Che, W., Nabende, J., Shutova, E., and Pilehvar, M.T. HFT: Half Fine-Tuning for Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Vienna, Austria, 27 July–1 August 2025.
20. Chiruzzo, L., Ritter, A., and Wang, L. Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, Albuquerque, NM, USA, 29 April–4 May 2025.
21. Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. (2023, January 1–5). Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda.
22. Qlora: Efficient finetuning of quantized llms
23. Schotthöfer, S., Zangrando, E., Ceruti, G., Tudisco, F., and Kusch, J. (2025, January 24). GeoLoRA: Geometric integration for parameter efficient fine-tuning. Proceedings of the Thirteenth International Conference on Learning Representations, Singapore.
24. A Primer in BERTology: What We Know About How BERT Works10.1162/tacl_a_00349 ↗
25. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M.S., and Love, J. (2024). Gemma: Open models based on gemini research and technology. arXiv.
26. Phi-2: The surprising power of small language models
27. Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., and Zhao, W. (2024). Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv.
28. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2019). SuperGLUE: A stickier benchmark for general-purpose language understanding systems. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, QC, Canada, 8–14 December 2019, Curran Associates Inc.