A Survey on Hardware Accelerators for Large Language Models
Source abstract
Abstract
Large language models (LLMs) have emerged as powerful tools for natural language processing tasks, revolutionizing the field with their ability to understand and generate human-like text. As the demand for more sophisticated LLMs continues to grow, there is a pressing need to address the computational challenges associated with their scale and complexity. This paper presents a comprehensive survey of hardware accelerators designed to enhance the performance and energy efficiency of large language models. By examining a diverse range of accelerators, including GPUs, FPGAs, and custom-designed architectures, we explore the landscape of hardware solutions tailored to meet the unique computational demands of LLMs. The survey encompasses an in-depth analysis of architecture, performance metrics, and energy efficiency considerations, providing valuable insights for researchers, engineers, and decision-makers aiming to optimize the deployment of LLMs in real-world applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Parallel Computing and Optimization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical and Electronics Engineering, University of West Attica, 12243 Egaleo, Greece
References (39)
Show source-provided references
1. Zhong, J., Liu, Z., and Chen, X. (2023). Transformer-based models and hardware acceleration analysis in autonomous driving: A survey. arXiv.
2. Hardware-friendly compression and hardware acceleration for transformer: A survey10.3934/era.2022192 ↗
3. Emani, M., Foreman, S., Sastry, V., Xie, Z., Raskar, S., Arnold, W., Thakur, R., Vishwanath, V., and Papka, M.E. (2023). A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators. arXiv.
4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv.
5. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
6. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2019). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv.10.18653/v1/W18-5446 ↗
7. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv.10.18653/v1/D16-1264 ↗
8. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2020). SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. arXiv.
9. Jang, H., Kim, J., Jo, J.E., Lee, J., and Kim, J. (2019, January 22–26). MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks. Proceedings of the 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA), Phoenix, AZ, USA.10.1145/3307650.3322214 ↗
10. Li, B., Pandey, S., Fang, H., Lyv, Y., Li, J., Chen, J., Xie, M., Wan, L., Liu, H., and Ding, C. (2020, January 10–11). FTRANS: Energy-Efficient Acceleration of Transformers Using FPGA. Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design, New York, NY, USA. ISLPED’20.10.1145/3370748.3406567 ↗
11. Lu, S., Wang, M., Liang, S., Lin, J., and Wang, Z. (2020, January 8–11). Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer. Proceedings of the 2020 IEEE 33rd International System-on-Chip Conference (SOCC), Virtual.10.1109/SOCC49529.2020.9524802 ↗
12. Khan, H., Khan, A., Khan, Z., Huang, L.B., Wang, K., and He, L. (2021). NPE: An FPGA-Based Overlay Processor for Natural Language Processing, Association for Computing Machinery. FPGA ’21.10.1145/3431920.3439477 ↗
13. Peng, H., Huang, S., Geng, T., Li, A., Jiang, W., Liu, H., Wang, S., and Ding, C. (2021, January 7–9). Accelerating Transformer-based Deep Learning Models on FPGAs using Column Balanced Block Pruning. Proceedings of the 2021 22nd International Symposium on Quality Electronic Design (ISQED), Santa Clara, CA, USA.10.1109/ISQED51717.2021.9424344 ↗
14. Hong, S., Moon, S., Kim, J., Lee, S., Kim, M., Lee, D., and Kim, J.Y. (2022). DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation. arXiv.10.1109/HCS55958.2022.9895626 ↗
15. Bai, Y., Zhou, H., Zhao, K., Chen, J., Yu, J., and Wang, K. (2023, January 8–11). Transformer-OPU: An FPGA-based Overlay Processor for Transformer Networks. Proceedings of the 2023 IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), Marina Del Rey, CA, USA.10.1109/FCCM57271.2023.00049 ↗
16. Tzanos, G., Kachris, C., and Soudris, D. (2022, January 2–3). Hardware Acceleration of Transformer Networks using FPGAs. Proceedings of the 2022 Panhellenic Conference on Electronics and Telecommunications (PACET), Tripolis, Greece.10.1109/PACET56979.2022.9976354 ↗
17. A Fast and Flexible FPGA-Based Accelerator for Natural Language Processing Neural Networks10.1145/3564606 ↗
18. Okubo, I., Sugiura, K., and Matsutani, H. (2024). A Cost-Efficient FPGA Implementation of Tiny Transformer Model using Neural ODE. arXiv.10.1109/IPDPSW59300.2023.00029 ↗
19. Choi, J., Li, H., Kim, B., Hwang, S., and Ahn, J.H. (2022, January 6–8). Accelerating Transformer Networks through Recomposing Softmax Layers. Proceedings of the 2022 IEEE International Symposium on Workload Characterization (IISWC), Austin, TX, USA.10.1109/IISWC55918.2022.00018 ↗
20. Wang, X., Wei, Y., Xiong, Y., Huang, G., Qian, X., Ding, Y., Wang, M., and Li, L. (2022, January 13–18). LightSeq2: Accelerated Training for Transformer-Based Models on GPUs. Proceedings of the SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, Dallas, TX, USA.10.1109/SC41404.2022.00043 ↗
21. He, B., and Hofmann, T. (2023). Simplifying Transformer Blocks. arXiv.
22. Yang, N., Ge, T., Wang, L., Jiao, B., Jiang, D., Yang, L., Majumder, R., and Wei, F. (2023). Inference with Reference: Lossless Acceleration of Large Language Models. arXiv.
23. Xia, H., Ge, T., Wang, P., Chen, S.Q., Wei, F., and Sui, Z. (2023). Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. arXiv.10.18653/v1/2023.findings-emnlp.257 ↗
24. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.B., Sifre, L., and Jumper, J. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv.
25. Belcak, P., and Wattenhofer, R. (2023). Exponentially Faster Language Modelling. arXiv.
26. Ham, T.J., Jung, S.J., Kim, S., Oh, Y.H., Park, Y., Song, Y., Park, J.H., Lee, S., Park, K., and Lee, J.W. (2020). A3: Accelerating Attention Mechanisms in Neural Networks with Approximation. arXiv.10.1109/HPCA47549.2020.00035 ↗
27. Ham, T.J., Lee, Y., Seo, S.H., Kim, S., Choi, H., Jung, S.J., and Lee, J.W. (2021, January 14–18). ELSA: Hardware-Software Co-design for Efficient, Lightweight Self-Attention Mechanism in Neural Networks. Proceedings of the 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), Valencia, Spain.10.1109/ISCA52012.2021.00060 ↗
28. Wang, H., Zhang, Z., and Han, S. (2021). SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning. arXiv.10.1109/HPCA51647.2021.00018 ↗
29. Lu, L., Jin, Y., Bi, H., Luo, Z., Li, P., Wang, T., and Liang, Y. (2021, January 18–22). Sanger: A Co-Design Framework for Enabling Sparse Attention Using Reconfigurable Architecture. Proceedings of the MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, New York, NY, USA. MICRO ’21.10.1145/3466752.3480125 ↗
30. Energon: Toward Efficient Acceleration of Transformers Using Dynamic Sparse Attention10.1109/TCAD.2022.3170848 ↗
31. H3D-Transformer: A Heterogeneous 3D (H3D) Computing Platform for Transformer Model Acceleration on Edge Devices10.1145/3649219 ↗
32. Guo, H., Peng, L., Zhang, J., Chen, Q., and LeCompte, T.D. (2020, January 18–21). ATT: A Fault-Tolerant ReRAM Accelerator for Attention-based Neural Networks. Proceedings of the 2020 IEEE 38th International Conference on Computer Design (ICCD), Los Alamitos, CA, USA.10.1109/ICCD50377.2020.00047 ↗
33. Yang, X., Yan, B., Li, H., and Chen, Y. (2020, January 2–5). ReTransformer: ReRAM-based Processing-in-Memory Architecture for Transformer Acceleration. Proceedings of the 2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD), Virtual.10.1145/3400302.3415640 ↗
34. Laguna, A.F., Kazemi, A., Niemier, M., and Hu, X.S. (2021, January 1–5). In-Memory Computing based Accelerator for Transformer Networks for Long Sequences. Proceedings of the 2021 Design, Automation and Test in Europe Conference and Exhibition (DATE), Virtual.10.23919/DATE51398.2021.9474146 ↗
35. Sridharan, S., Stevens, J.R., Roy, K., and Raghunathan, A. (2023). X-Former: In-Memory Acceleration of Transformers. arXiv.10.1109/TVLSI.2023.3282046 ↗
36. Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Mundo, C.C.D., Rastegari, M., and Farajtabar, M. (2024). LLM in a flash: Efficient Large Language Model Inference with Limited Memory. arXiv.10.18653/v1/2024.acl-long.678 ↗
37. Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. arXiv.
38. Shivakumar, P., and Jouppi, N. (2001). CACTI 3.0: An Integrated Cache Timing, Power, and Area Model, Western Research Laboratory.
39. The CUBLAS and CULA based GPU acceleration of adaptive finite element framework for bioluminescence tomography10.1364/OE.18.020201 ↗