Systematic Analysis of Low-Precision Training in Deep Neural Networks: Factors Influencing Matrix Computations
Source abstract
Abstract
As Deep Neural Networks (DNNs) continue to increase in complexity, the computational demands of their training have become a significant bottleneck. Low-precision training has emerged as a crucial strategy, wherein full-precision values are quantized to lower precisions, reducing computational overhead while aiming to maintain model accuracy. While prior research has primarily focused on minimizing quantization noise and optimizing performance for specific models and tasks, a comprehensive understanding of the general principles governing low-precision computations across diverse DNN architectures has been lacking. In this paper, we address this gap by systematically analyzing the factors that influence low-precision matrix computations, which are fundamental to DNN training. We investigate three critical factors—accumulation in matrix calculations, the frequency of element usage, and the depth of matrices within the model—and their impact on low-precision training. Through controlled experiments on standard models, as well as customized experiments designed to isolate individual factors, we derive several key insights: layers with higher accumulation and matrices with lower usage frequencies demonstrate greater tolerance to low-precision noise, without significantly compromising the stability of model training. Additionally, while the depth of matrices influences the stability of matrix operations to some extent, it does not have a noticeable effect on the overall training outcomes. Our findings contribute to the development of generalizable principles for low-precision training, offering a systematic framework applicable across various DNN architectures. We provide empirical evidence supporting the strategic allocation of training bit-widths based on the analyzed factors, thereby enhancing the efficiency and effectiveness of DNN training in resource-constrained environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Stochastic Gradient Optimization Techniques · Adversarial Robustness in Machine Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology, Changsha 410073, China
Funding
National Key R&D Program of China · National Natural Science Foundation of China
References (76)
Show source-provided references
1. Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M.W., and Keutzer, K. (2024). Ai and memory wall. arXiv.10.1109/MM.2024.3373763 ↗
2. Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettlemoyer, L. (2021, January 18–24). Base layers: Simplifying training of large, sparse models. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
3. Courbariaux, M., Bengio, Y., and David, J.P. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems, Curran Associates, Inc.
4. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2018, January 18–23). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00286 ↗
5. Lin, X., Zhao, C., and Pan, W. (2017). Towards accurate binary convolutional neural network. Advances in Neural Information Processing Systems, Curran Associates, Inc.
6. Yang, H., Duan, L., Chen, Y., and Li, H. (2021). BSQ: Exploring bit-level sparsity for mixed-precision neural network quantization. arXiv.
7. Bayesian bits: Unifying quantization and pruning
8. A statistical framework for low-bitwidth training of deep neural networks
9. Yang, C., Wu, Z., Chee, J., De Sa, C., and Udell, M. (2021). How Low Can We Go: Trading Memory for Error in Low-Precision Training. arXiv.
10. Chen, J., Zheng, L., Yao, Z., Wang, D., Stoica, I., Mahoney, M., and Gonzalez, J. (2021, January 18–24). Actnn: Reducing training memory footprint via 2-bit activation compressed training. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
11. Köster, U., Webb, T., Wang, X., Nassar, M., Bansal, A.K., Constable, W., Elibol, O., Gray, S., Hall, S., and Hornof, L. (2017). Flexpoint: An adaptive numerical format for efficient training of deep neural networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.
12. Fox, S., Rasoulinezhad, S., Faraone, J., Boland, D., and Leong, P. (2020, January 26–30). A block minifloat representation for training deep neural networks. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
13. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. (2016). Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv.
14. Han, S., Mao, H., and Dally, W.J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv.
15. Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. (2020, January 26–30). Once for All: Train One Network and Specialize it for Efficient Deployment. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
16. Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q., Wang, Y., and Mahoney, M. (2021, January 18–24). Hawq-v3: Dyadic neural network quantization. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
17. Banner, R., Nahshan, Y., and Soudry, D. (2019). Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems, Curran Associates, Inc.
18. Kim, M., and Smaragdis, P. (2016). Bitwise neural networks. arXiv.
19. Li, F., Liu, B., Wang, X., Zhang, B., and Yan, J. (2016). Ternary weight networks. arXiv.
20. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016, January 11–14). Xnor-net: Imagenet classification using binary convolutional neural networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_32 ↗
21. Uniq: Uniform noise injection for non-uniform quantization of neural networks10.1145/3444943 ↗
22. Statistical theory of quantization10.1109/19.492748 ↗
23. Binary neural networks: A survey10.1016/j.patcog.2020.107281 ↗
24. Wu, S., Li, G., Chen, F., and Shi, L. (2018). Training and inference with integers in deep neural networks. arXiv.
25. Zhuang, B., Shen, C., Tan, M., Liu, L., and Reid, I. (2018, January 18–22). Towards effective low-bitwidth convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00826 ↗
26. Martinez, B., Yang, J., Bulat, A., and Tzimiropoulos, G. (2020). Training binary neural networks with real-to-binary convolutions. arXiv.
27. Chakrabarti, A., and Moseley, B. (2019). Backprop with approximate activations for memory-efficient network training. Advances in Neural Information Processing Systems, Curran Associates, Inc.
28. Yang, G., Zhang, T., Kirichenko, P., Bai, J., Wilson, A.G., and De Sa, C. (2019, January 9–15). SWALP: Stochastic weight averaging in low precision training. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
29. Choi, J., Wang, Z., Venkataramani, S., Chuang, P.I.J., Srinivasan, V., and Gopalakrishnan, K. (2018). Pact: Parameterized clipping activation for quantized neural networks. arXiv.
30. Das, D., Mellempudi, N., Mudigere, D., Kalamkar, D., Avancha, S., Banerjee, K., Sridharan, S., Vaidyanathan, K., Kaul, B., and Georganas, E. (2018). Mixed precision training of convolutional neural networks using integer operations. arXiv.
31. Zhang, D., Yang, J., Ye, D., and Hua, G. (2018, January 8–14). Lq-nets: Learned quantization for highly accurate and compact deep neural networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_23 ↗
32. Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., and Modha, D.S. (2019). Learned step size quantization. arXiv.
33. Gholami, A., Kwon, K., Wu, B., Tai, Z., Yue, X., Jin, P., Zhao, S., and Keutzer, K. (2018, January 18–22). Squeezenext: Hardware-aware neural network design. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00215 ↗
34. Banner, R., Hubara, I., Hoffer, E., and Soudry, D. (2018). Scalable methods for 8-bit training of neural networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.
35. Zhu, F., Gong, R., Yu, F., Liu, X., Wang, Y., Li, Z., Yang, X., and Yan, J. (2020, January 14–19). Towards unified int8 training for convolutional neural network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00204 ↗
36. Lee, S., Park, J., and Jeon, D. (2021, January 3–7). Toward Efficient Low-Precision Training: Data Format Optimization and Hysteresis Quantization. Proceedings of the International Conference on Learning Representations, Virtual.
37. Training high-performance and large-scale deep neural networks with full 8-bit integers10.1016/j.neunet.2019.12.027 ↗
38. Wang, N., Choi, J., Brand, D., Chen, C.Y., and Gopalakrishnan, K. (2018). Training deep neural networks with 8-bit floating point numbers. Advances in Neural Information Processing Systems, Curran Associates, Inc.
39. Xi, H., Li, C., Chen, J., and Zhu, J. (2023). Training Transformers with 4-bit Integers. arXiv.
40. Han, R., Si, M., Demmel, J., and You, Y. (2021, January 27). Dynamic scaling for low-precision learning. Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Virtual.10.1145/3437801.3441624 ↗
41. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., and Venkatesh, G. (2017). Mixed precision training. arXiv.
42. Sun, X., Choi, J., Chen, C.Y., Wang, N., Venkataramani, S., Srinivasan, V.V., Cui, X., Zhang, W., and Gopalakrishnan, K. (2019). Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.
43. Ultra-low precision 4-bit training of deep neural networks
44. Ma, Z., He, J., Qiu, J., Cao, H., Wang, Y., Sun, Z., Zheng, L., Wang, H., Tang, S., and Zheng, T. (2022, January 2–6). BaGuaLu: Targeting brain scale pretrained models with over 37 million cores. Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Seoul, Republic of Korea.10.1145/3503221.3508417 ↗
45. Drumond, M., Lin, T., Jaggi, M., and Falsafi, B. (2018). Training dnns with hybrid block floating point. Advances in Neural Information Processing Systems, Curran Associates, Inc.
46. Fu, Y., Guo, H., Li, M., Yang, X., Ding, Y., Chandra, V., and Lin, Y.L. (2021, January 3–7). CPT: Efficient Deep Neural Network Training via Cyclic Precision. Proceedings of the 9th International Conference on Learning Representations 2021 (ICLR 2021), Virtual.
47. Wikipedia Contributors (2024, October 22). Propagation of Uncertainty—Wikipedia, The Free Encyclopedia. Available online: https://en.wikipedia.org/wiki/Propagation_of_uncertainty.
48. Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-Dickstein, J. (2017, January 6–11). On the expressive power of deep neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
49. He, C., Li, S., Soltanolkotabi, M., and Avestimehr, S. (2021). Pipetransformer: Automated elastic pipelining for distributed training of transformers. arXiv.
50. Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. (2017). Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in Neural Information Processing Systems, Curran Associates, Inc.
51. Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., and Courville, A. (2019, January 9–15). On the spectral bias of neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
52. Xu, Z.Q.J., Zhang, Y., Luo, T., Xiao, Y., and Ma, Z. (2019). Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv.
53. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
54. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
55. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–22). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
56. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images. [Master’s Thesis, University of Tront].
57. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
58. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, Curran Associates, Inc.
59. Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2016, January 2–4). Pointer Sentinel Mixture Models. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico.
60. Graves, A., and Graves, A. (2012). Long short-term memory. Supervised Sequence Labelling with Recurrent Neural Networks, Springer.10.1007/978-3-642-24797-2 ↗
61. Building a Large Annotated Corpus of English: The Penn Treebank
62. Wang, X., Yu, F., Dou, Z.Y., Darrell, T., and Gonzalez, J.E. (2018, January 8–14). Skipnet: Learning dynamic routing in convolutional networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01261-8_25 ↗
63. Baevski, A., and Auli, M. (May, January 30). Adaptive Input Representations for Neural Language Modeling. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
64. Merity, S., Keskar, N.S., and Socher, R. (2017). Regularizing and optimizing LSTM language models. arXiv.
65. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
66. Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. (2023, January 6–10). Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.102 ↗
67. Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., and Zhu, Y. (2023, January 17–21). OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization. Proceedings of the 50th Annual International Symposium on Computer Architecture, Orlando, FL, USA.10.1145/3579371.3589038 ↗
68. Guo, C., Zhang, C., Leng, J., Liu, Z., Yang, F., Liu, Y., Guo, M., and Zhu, Y. (2022, January 1–5). Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization. Proceedings of the 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), Chicago, IL, USA.10.1109/MICRO56248.2022.00095 ↗
69. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023, January 1–5). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda.
70. Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank10.1227/neu.0000000000002551 ↗
71. Erhan, D., Courville, A., Bengio, Y., and Vincent, P. (2010, January 13–15). Why does unsupervised pre-training help deep learning?. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia Laguna Resort, Sardinia, Italy.
72. Nichol, A.Q., and Dhariwal, P. (2021, January 18–24). Improved denoising diffusion probabilistic models. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
73. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
74. Liu, S.y., Liu, Z., Huang, X., Dong, P., and Cheng, K.T. (2023, January 6–10). LLM-FP4: 4-Bit Floating-Point Quantized Transformers. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.39 ↗
75. Fu, F., Hu, Y., He, Y., Jiang, J., Shao, Y., Zhang, C., and Cui, B. (2020, January 13–18). Don’t waste your bits! Squeeze activations and gradients for deep neural networks via tinyscript. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
76. Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv.