SQ-LoRA: Memory-Efficient Language Model Compression Through Stable-Rank-Guided Quantization for Edge Computing Applications
Source abstract
Abstract
The deployment of transformer-based language models on resource-constrained edge devices presents fundamental challenges in computational efficiency and memory utilization. We introduce SQ-LoRA (Stable-rank Quantized Low-Rank Adaptation), a theoretically grounded compression framework that achieves unprecedented efficiency through the synergistic integration of adaptive low-rank decomposition, hardware-accelerated structured sparsity, and intelligent hybrid quantization. Our primary contribution establishes the first rigorous mathematical connection between the matrix stable rank and optimal LoRA rank selection, formalized in Theorem I, which provides bounded approximation guarantees. SQ-LoRA implements: (1) adaptive rank allocation via stable-rank analysis to automatically determine layer-wise compression ratios; (2) 4:8 structured sparsity patterns, enabling 2× hardware acceleration on modern edge processors; and (3) a three-tier quantization scheme that combines 4-bit NormalFloat storage with selective 3-bit/8-bit precision to preserve outliers. A comprehensive evaluation on four diverse natural language processing (NLP) benchmarks demonstrates that SQ-LoRA achieves a 320 MB memory footprint (96.7% reduction) and a 10 ms inference latency (91.7% improvement), and maintains 82.0% average accuracy (within 0.15% of the full model). Statistical significance testing (p < 0.001) confirms its superiority over state-of-the-art methods. This framework enables the deployment of sophisticated language models on devices with 2 GB of RAM, advancing practical edge-AI applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Stochastic Gradient Optimization Techniques · Advanced Neural Network Applications · Parallel Computing and Optimization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Iğdır University, Iğdır 76000, Türkiye
Funding
Scientific and Technological Research Council of Türkiye
References (34)
Show source-provided references
1. Attention is all you need
2. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
3. (2026, February 17). IDC, Worldwide Edge Spending Guide: IDC Estimates Global Spending on Edge Computing to Grow at 13.8% Reaching Nearly $380 Billion by 2028, IDC Press Release. Available online: https://www.idc.com/getdoc.jsp?containerId=prUS53261225.
4. Edge computing for autonomous driving: Opportunities and challenges10.1109/JPROC.2019.2915983 ↗
5. Edge intelligence: Paving the last mile of artificial intelligence with edge computing10.1109/JPROC.2019.2918951 ↗
6. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 23–27). LoRA: Low-rank adaptation of large language models. Proceedings of the International Conference on Learning Representations, Rio de Janeiro, Brazil.
7. A survey of quantization methods for efficient neural network inference
8. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023, January 10–16). QLoRA: Efficient finetuning of quantized LLMs. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
9. Sparsity in deep learning: Pruning and growth for efficient inference and training
10. Han, S., Mao, H., and Dally, W.J. (2016, January 2–4). Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico.
11. Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. (2023, January 1–5). Adaptive budget allocation for parameter-efficient fine-tuning. Proceedings of the 11th International Conference on Learning Representations, Kigali, Rwanda.
12. Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. (2023, January 2–6). DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Dubrovnik, Croatia.10.18653/v1/2023.eacl-main.239 ↗
13. Aghajanyan, A., Gupta, S., and Zettlemoyer, L. (2021, January 1–6). Intrinsic dimensionality explains the effectiveness of language model fine-tuning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), Online.10.18653/v1/2021.acl-long.568 ↗
14. Wang, F., Zhang, X., Li, Z., and Gao, Z. (2024). LoRA-GA: Low-rank adaptation with gradient approximation. arXiv.
15. Post training 4-bit quantization of convolutional networks for rapid-deployment
16. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2018, January 18–22). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE Conference on Computer Vision and pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00286 ↗
17. Gpt3. int8(): 8-bit matrix multiplication for transformers at scale
18. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023, January 1–5). GPTQ: Accurate post-training quantization for generative pre-trained transformers. Proceedings of the 11th International Conference on Learning Representations, Kigali, Rwanda.
19. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. (2024, January 13–16). AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. Proceedings of the 7th Annual Conference on Machine Learning and Systems (MLSys), Santa Clara, CA, USA.
20. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers
21. Channel permutations for N:M sparsity
22. Zhou, A., Ma, Y., Zhu, J., Liu, J., Zhang, Z., Yuan, K., Sun, W., and Li, H. (2021, January 3–7). Learning N:M fine-grained structured sparse neural networks from scratch. Proceedings of the 9th International Conference on Learning, Online.
23. Mishra, A., Latorre, J.A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. (2021). Accelerating sparse deep neural networks. arXiv.
24. LLM-Pruner: On the structural pruning of large language models
25. Dou, S., Zhou, E., Liu, Y., Gao, S., Zhao, J., Shen, W., Zhou, Y., Xi, Z., Wang, X., and Fan, X. (2024). LORAMOE: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. arXiv.
26. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. (2023, January 23–29). High-throughput generative inference of large language models with a single GPU. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
27. Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C.F., Cheng, K.-T., and Chen, M.-H. (2024, January 21–27). DoRA: Weight-decomposed low-rank adaptation. Proceedings of the 41st International Conference on Machine Learning (ICML), Vienna, Austria.
28. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M.W., and Keutzer, K. (2024, January 21–27). Squeezellm: Dense-and-sparse quantization. Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria.
29. Evaluating the effectiveness of different machine learning approaches for sentiment classification
30. Bravo-Marquez, J.M.M.F., Salameh, M., and Kiritchenko, S. (2018, January 5–6). SemEval-2018 task 1: Affect in tweets. Proceedings of the 12th International Workshop on Semantic Evaluation, New Orleans, LA, USA.10.18653/v1/S18-1001 ↗
31. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., and Potts, C. (2013, January 18–21). Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.10.18653/v1/D13-1170 ↗
32. Clark, C., Lee, K., Chang, M.W., Kwiatkowski, T., Collins, M., and Toutanova, K. (2019, January 2–7). BoolQ: Exploring the surprising difficulty of natural yes/no questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
33. Williams, A., Nangia, N., and Bowman, S. (2018, January 1–6). A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.10.18653/v1/N18-1101 ↗
34. Frankle, J., and Carbin, M. (2019, January 6–9). The lottery ticket hypothesis: Finding sparse, trainable neural networks. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.