Optimizing Fine-Tuning in Quantized Language Models: An In-Depth Analysis of Key Variables
Source abstract
Abstract
Large-scale Language Models (LLMs) have achieved significant breakthroughs in Natural Language Processing (NLP), driven by the pre-training and fine-tuning paradigm. While this approach allows models to specialize in specific... | Find, read and cite all the research you need on Tech Science Press
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (50)
Show source-provided references
1. J. Achiam et al., “GPT-4 technical report,” 2023, arXiv:2303.08774.
2. S. Bubeck et al., “Sparks of artificial general intelligence: Early experiments with GPT-4,” 2023, arXiv:2303.12712.
3. Bloom: A 176b-parameter open-access multilingual language model
4. H. Touvron et al., “LLaMA: Open and efficient foundation language models,” 2023, arXiv:2302.13971.
5. J. Wei et al., “Emergent abilities of large language models,” 2022, arXiv:2206.07682.
6. Language models are few-shot learners
7. J. Devlin, M. -W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2018, arXiv:1810.04805.
8. W. X. Zhao et al., “A survey of large language models,” 2023, arXiv:2303.18223.
9. X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” 2021, arXiv:2101.00190.
10. Parameter-efficient transfer learning for NLP
11. T. Dettmers et al., “SpQR: A sparse-quantized representation for near-lossless llm weight compression,” 2023, arXiv:2306.03078.
12. J. Lin et al., “AWQ: Activation-aware weight quantization for llm compression and acceleration,” 2023, arXiv:2306.00978.
13. SmoothQuant: Accurate and efficient post-training quantization for large language models
14. Outlier suppression: Pushing the limit of low-bit transformer language models
15. E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “LoRA: Low-rank adaptation of large language models,” 2021, arXiv:2106.09685.
16. S. Hayou, N. Ghosh, and B. Yu, “LoRA+: Efficient low rank adaptation of large models,” 2024, arXiv:2402.12354.
17. B. Zi, X. Qi, L. Wang, J. Wang, K. -F. Wong and L. Zhang, “Delta-LoRA: Fine-tuning high-rank parameters with the delta of low-rank matrices,” 2023, arXiv:2309.02411.
18. QLoRA: Efficient finetuning of quantized LLMs
19. A. Shen, Q. Wang, Z. Lai, X. Li, and D. Li, “Accurate and efficient fine-tuning of quantized large language models through optimal balance,” 2024, arXiv:2407.17029.
20. H. Qin et al., “Accurate LoRA-finetuning quantization of LLMs via information retention,” 2024, arXiv:2402.05445.
21. QA-LORA: Quantization-aware low-rank adaptation of large language models
22. S. Liu et al., “The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training,” 2022, arXiv:2202.02643.
23. Y. Fu et al., “CPT: Efficient deep neural network training via cyclic precision,” 2021, arXiv:2101.09868.
24. How transferable are features in deep neural networks?
25. Visualizing and understanding convolutional networks
26. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale
27. The case for 4-bit precision: k-bit inference scaling laws
28. J. Kaplan et al., “Scaling laws for neural language models,” 2020, arXiv:2001.08361.
29. T. Henighan et al., “Scaling laws for autoregressive generative modeling,” 2020, arXiv:2010.14701.
30. Quantization and training of neural networks for efficient integer-arithmetic-only inference
31. S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” 2015, arXiv:1510.00149.
32. Once for all: Train one network and specialize it for efficient deployment
33. GPTQ: Accurate post-training quantization for generative pre-trained transformers
34. X. Wu, Z. Yao, and Y. He, “ZeroQuant-FP: A leap forward in llms post-training W4A8 quantization using floating-point formats,” 2023, 2023, arXiv:2307.09782.
35. Dettmers · Neural Information Processing Systems (2022) · 2022 · 35 · 30318
36. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science10.1038/s41467-018-04316-3 ↗
37. Rigging the lottery: Making all tickets winners
38. S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen and Y. Zou, “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” 2016, arXiv:1606.06160.
39. BaGuaLu: Targeting brain scale pretrained models with over 37 million cores
40. C. Yang, Z. Wu, J. Chee, C. De Sa, and M. Udell, “How low can we go: Trading memory for error in low-precision training,” 2021, arXiv:2106.09686.
41. T. Jiang et al., “MoRA: High-rank updating for parameter-efficient fine-tuning,” 2024, arXiv:2405.12130.
42. H. Touvron et al., “LLaMA 2: Open foundation and fine-tuned chat models,” 2023, arXiv:2307.09288.
43. R. Taori et al., “Stanford alpaca: An instruction-following llama model,” 2023. Accessed: Aug. 15, 2024. [Online]. Available: https://github.com/tatsulab/stanford_alpaca
44. D. Hendrycks et al., “Measuring massive multitask language understanding,” 2020, arXiv:2009.03300.
45. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “HellaSwag: Can a machine really finish your sentence?” 2019, arXiv:1905.07830.10.18653/v1/P19-1472 ↗
46. PIQA: Reasoning about physical commonsense in natural language
47. WinoGrande: An adversarial winograd schema challenge at scale10.1145/3474381 ↗
48. P. Clark et al., “Think you have solved question answering? try ARC, the AI2 reasoning challenge,” 2018, arXiv:1803.05457.
49. C. Clark, K. Lee, M. -W. Chang, T. Kwiatkowski, M. Collins and K. Toutanova, “BoolQ: Exploring the surprising difficulty of natural yes/no questions,” 2019, arXiv:1905.10044.
50. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? A new dataset for open book question answering,” 2018, arXiv:1809.02789.10.18653/v1/D18-1260 ↗