Analyzing LLAMA3 Performance on Classification Task Using LoRA and QLoRA Techniques
Source abstract
Abstract
Large language models (LLMs), consisting of billions and trillions of parameters, have demonstrated exceptional ability in natural language understanding (NLU) and natural language generation (NLG) tasks. Increases in their numbers of parameters and model sizes have resulted in better performance and accuracy. However, models with such enormous numbers of parameters incur significant computational costs and resources, making them challenging to fine tune and adapt to a specific downstream task. Several parameter-efficient fine-tuning (PEFT) techniques have been proposed to address this issue. This study demonstrates the improvement obtained over the base LLaMA3-8B model using two prominent PEFT techniques: LoRA and QLoRA. We use the sequence classification task of sentiment analysis to conduct the experiments. Additionally, we analyze the effects of hyperparameter adjustments (r and α) on the model’s performance. We examine the tradeoff between efficiency and memory savings obtained using the quantized LoRA (QLoRA) technique. We also investigate and compare the performance changes of LoRA and QLoRA techniques obtained after adapting to attention layers (query, key, value, and project) to all the linear layers during fine tuning. We report the findings of our work along with limitations and future directions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Automated Systems · Context-Aware Activity Recognition Systems · Fuzzy Logic and Control Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computing, Grand Valley State University, Allendale, MI 49401, USA
Computer Science Department, East Carolina University, Greenville, NC 27858, USA
References (33)
Show source-provided references
1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2025, February 15). Attention Is All You Need. Advances in Neural Information Processing Systems. 2017. Available online: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
2. Exploring the limits of transfer learning with a unified text-to-text transformer
3. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
4. Language models are few-shot learners
5. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
6. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
7. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). Opt: Open pre-trained transformer language models. arXiv.
8. Chain-of-thought prompting elicits reasoning in large language models
9. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. (2022). A survey on in-context learning. arXiv.
10. Wang, T., Roberts, A., Hesslow, D., Le Scao, T., Chung, H.W., Beltagy, I., Launay, J., and Raffel, C. (2022, January 28–30). What language model architecture and pretraining objective works best for zero-shot generalization?. Proceedings of the International Conference on Machine Learning, PMLR, Online.
11. (2024, November 15). Introducing Meta Llama 3: The Most Capable Openly Available LLM to Date. Available online: https://huggingface.co/meta-llama/Meta-Llama-3-8B.
12. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019, January 10–15). Parameter-efficient transfer learning for NLP. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
13. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.M., Rothchild, D., So, D., Texier, M., and Dean, J. (2021). Carbon emissions and large neural network training. arXiv.
14. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv.
15. Qlora: Efficient finetuning of quantized llms
16. Li, C., Farkhoor, H., Liu, R., and Yosinski, J. (2018). Measuring the intrinsic dimension of objective landscapes. arXiv.
17. Aghajanyan, A., Zettlemoyer, L., and Gupta, S. (2020). Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv.10.18653/v1/2021.acl-long.568 ↗
18. Zhang, X., Zhao, J., and LeCun, Y. (2025, February 15). Character-level Convolutional Networks for Text Classification, Advances in Neural Information Processing Systems, 28, NIPS 2015. Available online: https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html.
19. Yu, Z., and Ananiadou, S. (2024, January 12–16). Neuron-level knowledge attribution in large language models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.191 ↗
20. Zheng, Z., Wang, Y., Huang, Y., Song, S., Yang, M., Tang, B., Xiong, F., and Li, Z. (2024). Attention heads of large language models: A survey. arXiv.10.1016/j.patter.2025.101176 ↗
21. Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. (2023). AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv.
22. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning
23. Hyeon-Woo, N., Ye-Bin, M., and Oh, T.H. (2021). Fedpara: Low-rank hadamard product for communication-efficient federated learning. arXiv.
24. Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. (2022). Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv.10.18653/v1/2023.eacl-main.239 ↗
25. Liu, S.Y., Wang, C.Y., Yin, H., Molchanov, P., Wang, Y.C.F., Cheng, K.T., and Chen, M.H. (2024). Dora: Weight-decomposed low-rank adaptation. arXiv.
26. Chavan, A., Liu, Z., Gupta, D., Xing, E., and Shen, Z. (2023). One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv.
27. Edalati, A., Tahaei, M., Kobyzev, I., Nia, V.P., Clark, J.J., and Rezagholizadeh, M. (2022). Krona: Parameter efficient tuning with kronecker adapter. arXiv.
28. Compacter: Efficient low-rank hypercomplex adapter layers
29. Newman, B., Choubey, P.K., and Rajani, N. (2021). P-adapters: Robustly extracting factual information from language models with diverse prompts. arXiv.
30. Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. arXiv.10.18653/v1/2021.emnlp-main.243 ↗
31. Li, X.L., and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv.10.18653/v1/2021.acl-long.353 ↗
32. Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., and Qiao, Y. (2023). Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv.
33. GPT Understands, Too10.1016/j.aiopen.2023.08.012 ↗