A Review of the Optimal Design of Neural Networks Based on FPGA
Source abstract
Abstract
Deep learning based on neural networks has been widely used in image recognition, speech recognition, natural language processing, automatic driving, and other fields and has made breakthrough progress. FPGA stands out in the field of accelerated deep learning with its advantages such as flexible architecture and logic units, high energy efficiency ratio, strong compatibility, and low delay. In order to track the latest research results of neural network optimization technology based on FPGA in time and to keep abreast of current research hotspots and application fields, the related technologies and research contents are reviewed. This paper introduces the development history and application fields of some representative neural networks and points out the importance of studying deep learning technology, as well as the reasons and advantages of using FPGA to accelerate deep learning. Several common neural network models are introduced. Moreover, this paper reviews the current mainstream FPGA-based neural network acceleration technology, method, accelerator, and acceleration framework design and the latest research status, pointing out the current FPGA-based neural network application facing difficulties and the corresponding solutions, as well as prospecting the future research directions. We hope that this work can provide insightful research ideas for the researchers engaged in the field of neural network acceleration based on FPGA.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Neural Networks and Applications · Embedded Systems and FPGA Applications · Industrial Vision Systems and Defect Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Automation and Information Engineering, Sichuan University of Science and Engineering, Yibin 644000, China
Artificial Intelligence Key Laboratory of Sichuan Province, Sichuan University of Science and Engineering, Yibin 644000, China
Funding
National Natural Science Foundation of China · Sichuan Science and Technology Program · Innovation Fund of Chinese Universities · Sichuan University of Science and Engineering Talent Introduction Project · Postgraduate Innovation Fund Project of Sichuan University of Science and Engineering
References (100+)
Show source-provided references
1. Deep learning based multi-source localization with source splitting and its effectiveness in multi-talker speech recognition10.1016/j.csl.2022.101360 ↗
2. Deep learning based assistive technology on audio visual speech recognition for hearing impaired
3. A model of speech recognition for hearing-impaired listeners based on deep learning10.1121/10.0009411 ↗
4. Temporal signals to images: Monitoring the condition of industrial assets with deep learning image processing algorithms
5. A review on deep learning in medical image analysis10.1007/s13735-021-00218-1 ↗
6. Deep learning in optical metrology: A review
7. An introduction to deep learning in natural language processing: Models, techniques, and tools10.1016/j.neucom.2021.05.103 ↗
8. Razumovskaia, E., Glavaš, G., Majewska, O., Ponti, E.M., and Vulic, I. (2022, January 22–27). Natural Language Processing for Multilingual Task-Oriented Dialogue. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, Dublin, Ireland.10.18653/v1/2022.acl-tutorials.8 ↗
9. Li, B., Hou, Y., and Che, W. (2022). Data Augmentation Approaches in Natural Language Processing: A Survey, AI Open.10.1016/j.aiopen.2022.03.001 ↗
10. Hu, Y., Liu, Y., and Liu, Z. (2022, January 7–9). A Survey on Convolutional Neural Network Accelerators: GPU, FPGA and ASIC. Proceedings of the 2022 14th International Conference on Computer Research and Development (ICCRD), Shenzhen, China.10.1109/ICCRD54409.2022.9730377 ↗
11. A survey on hardware accelerators and optimization techniques for RNNs10.1016/j.sysarc.2020.101839 ↗
12. A survey of hardware architectures for generative adversarial networks10.1016/j.sysarc.2021.102227 ↗
13. Review on FPGA-Based Accelerators in Deep Learning
14. Development, Implementation and Prospect of FPGA-Based Deep Neural Networks
15. A logical calculus of the ideas immanent in nervous activity10.1007/BF02478259 ↗
16. Turing, A. (2004). Intelligent Machinery (1948), B. Jack Copeland.10.1093/oso/9780198250791.003.0016 ↗
17. Hebb, D.O. (2005). The Organization of Behavior: A Neuropsychological Theory, Psychology Press.10.4324/9781410612403 ↗
18. The perceptron: A probabilistic model for information storage and organization in the brain10.1037/h0042519 ↗
19. Minsky, M., and Papert, S.A. (2017). Perceptrons, Reissue of the 1988 Expanded Edition with a New Foreword by Léon Bottou: An Introduction to Computational Geometry, MIT Press.10.7551/mitpress/11301.001.0001 ↗
20. Backpropagation through time: What it does and how to do it10.1109/5.58337 ↗
21. Fukushima, K., and Miyake, S. (1982). Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. Competition and Cooperation in Neural Nets, Springer.10.1007/978-3-642-46466-9_18 ↗
22. Neural networks and physical systems with emergent collective computational abilities10.1073/pnas.79.8.2554 ↗
23. A learning algorithm for Boltzmann machines
24. Learning representations by back-propagating errors10.1038/323533a0 ↗
25. Backpropagation applied to handwritten zip code recognition10.1162/neco.1989.1.4.541 ↗
26. Reducing the dimensionality of data with neural networks10.1126/science.1127647 ↗
27. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
28. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
29. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
30. Generative adversarial networks10.1145/3422622 ↗
31. Sun, Y., Wang, X., and Tang, X. (2014, January 23–28). Deep learning face representation from predicting 10,000 classes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA.10.1109/CVPR.2014.244 ↗
32. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA.10.1109/CVPR.2014.81 ↗
33. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
34. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, faster, stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
35. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
36. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
37. Li, C., Li, L., Jiang, H., Wenig, K., Geng, Y., Li, L., Ke, Z., Li, Q., Cheng, M., and Nie, W. (2022). YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications. arXiv.
38. Wang, C.Y., Bochkovskiy, A., and Liao, H.Y.M. (2022). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv.10.1109/CVPR52729.2023.00721 ↗
39. Hyperspectral Image Classification Using CNN-Enhanced Multi-Level Haar Wavelet Features Fusion Network
40. A transfer learning-based approach with deep cnn for covid-19-and pneumonia-affected chest x-ray image classification10.1007/s42979-021-00881-5 ↗
41. Breast cancer image classification using transfer learning and convolutional neural network
42. Han, G., Huang, S., Ma, J., and He, Y. (February, January 1). Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
43. Ramachandra, A.C. (2022). Real Time Object Detection System with YOLO and CNN Models: A Review. arXiv.
44. Semantic segmentation of land cover from high resolution multispectral satellite images by spectral-spatial convolutional neural network10.1080/10106049.2020.1734871 ↗
45. Valdez-Rodríguez, J.E., Calvo, H., Felipe-Riverón, E., and Moreno-Armendariz, M.A. (2022). Improving Depth Estimation by Embedding Semantic Segmentation: A Hybrid CNN Model. Sensors, 22.10.3390/s22041669 ↗
46. Nguyen, C., Asad, Z., Deng, R., and Huo, Y. (2022, January 14–16). Evaluating transformer-based semantic segmentation networks for pathological image segmentation. Proceedings of the Medical Imaging 2022: Image Processing, Tianjin, China.10.1117/12.2611177 ↗
47. Sağlam, S., Tat, F., and Bayar, S. (2019, January 27–29). FPGA Implementation of CNN Algorithm for Detecting Malaria Diseased Blood Cells. Proceedings of the 2019 International Symposium on Advanced Electrical and Communication Technologies (ISAECT), Rome, Italy.10.1109/ISAECT47714.2019.9069724 ↗
48. Zhang, Q. (2020). Application of CNN Optimization Design Based on APSOC in the Classification of Congenital Heart Disease. [Master’s Thesis, Yunnan University].
49. Image recognition of CT diagnosis for cholangiocarcinoma treatment based on FPGA processor and neural network10.1016/j.micpro.2020.103645 ↗
50. MRI-based brain tumor segmentation using FPGA-accelerated neural network10.1186/s12859-021-04347-6 ↗
51. Liu, H., Panahi, A., Andrews, D., and Nelson, A. (2020, January 9–11). An FPGA-Based Upper-Limb Rehabilitation Device for Gesture Recognition and Motion Evaluation Using Multi-Task Recurrent Neural Networks. Proceedings of the 2020 International Conference on Field-Programmable Technology (ICFPT), Maui, HI, USA.10.1109/ICFPT51103.2020.00054 ↗
52. Wang, C. (2020). Implementation and Verification of CNN Based on FPGA. [Ph.D. Thesis, Hebei University].
53. Design and Implementation of Accelerator for Aircrafts Key Points Detection Based on FPGA
54. Ferreira, J.C., and Fonseca, J. (December, January 30). An FPGA implementation of a long short-term memory neural network. Proceedings of the 2016 International Conference on ReConFigurable Computing and FPGAs (ReConFig), Cancun, Mexico.10.1109/ReConFig.2016.7857151 ↗
55. Guan, Y., Yuan, Z., Sun, G., and Cong, J. (2017, January 16–19). FPGA-based accelerator for long short-term memory recurrent neural networks. Proceedings of the 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC), Chiba, Japan.10.1109/ASPDAC.2017.7858394 ↗
56. Zhang, Y., Wang, C., Gong, L., Lu, Y., Sun, F., Xu, C., Li, X., and Zhou, X. (2017, January 12–15). Implementation and optimization of the accelerator based on FPGA hardware for LSTM network. Proceedings of the 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA/IUCC), Guangzhou, China.10.1109/ISPA/IUCC.2017.00098 ↗
57. Zhang, Y., Wang, C., Gong, L., Lu, Y., Sun, F., Xu, C., Li, X., and Zhou, X. (2017, January 5–8). A power-efficient accelerator based on FPGAs for LSTM network. Proceedings of the 2017 IEEE International Conference on Cluster Computing (CLUSTER), Honolulu, HI, USA.10.1109/CLUSTER.2017.45 ↗
58. Han, S., Kang, J., Mao, H., Hu, Y., Li, X., Li, Y., Xie, D., Luo, H., Yao, S., and Wang, Y. (2017, January 22–24). Ese: Efficient speech recognition engine with sparse lstm on fpga. Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, CA, USA.
59. Li, Z., Ding, C., Wang, S., Wen, W., Zhou, Y., Liu, C., Qiu, Q., Xu, W., Lin, X., and Qian, X. (2019, January 16–20). E-RNN: Design optimization for efficient recurrent neural networks in FPGAs. Proceedings of the 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA), Washington, DC, USA.10.1109/HPCA.2019.00028 ↗
60. Zheng, Y., Yang, H., Huang, Z., Li, T., and Jia, Y. (2019, January 9–13). A high energy-efficiency FPGA-based LSTM accelerator architecture design by structured pruning and normalized linear quantization. Proceedings of the 2019 International Conference on Field-Programmable Technology (ICFPT), Tianjin, China.10.1109/ICFPT47387.2019.00045 ↗
61. FiC-RNN: A multi-FPGA acceleration framework for deep recurrent neural networks10.1587/transinf.2020PAP0003 ↗
62. EdgeDRNN: Recurrent neural network accelerator for edge inference10.1109/JETCAS.2020.3040300 ↗
63. Kim, J., Kim, J., and Kim, T.H. (2021). AERO: A 1.28 MOP/s/LUT reconfigurable inference processor for recurrent neural networks in a resource-limited FPGA. Electronics, 10.10.3390/electronics10111249 ↗
64. A CPU-FPGA Heterogeneous Acceleration System for Scene Text Detection Network
65. Spartus: A 9.4 top/s fpga-based lstm accelerator exploiting spatio-temporal sparsity
66. Yazdanbakhsh, A., Brzozowski, M., Khaleghi, B., Ghodrati, S., Samadi, K., Kim, N.S., and Esmaeilzadeh, H. (May, January 29). Flexigan: An end-to-end solution for fpga acceleration of generative adversarial networks. Proceedings of the 2018 IEEE 26th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), Boulder, CO, USA.10.1109/FCCM.2018.00019 ↗
67. Chang, J.W., Ahn, S., Kang, K.W., and Kang, S.J. (2020, January 13–16). Towards design methodology of efficient fast algorithms for accelerating generative adversarial networks on FPGAs. Proceedings of the 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), Beijing, China.10.1109/ASP-DAC47756.2020.9045214 ↗
68. Shi, X.P. (2019). Research on the Infrared Image Enhancement Based on Generative Adversarial Networks. [Master’s Thesis, Tianjin University].
69. Danopoulos, D., Anagnostopoulos, K., Kachris, C., and Soudris, D. (2021, January 5–7). FPGA Acceleration of Generative Adversarial Networks for Image Reconstruction. Proceedings of the 2021 10th International Conference on Modern Circuits and Systems Technologies (MOCAST), Thessaloniki, Greece.10.1109/MOCAST52088.2021.9493361 ↗
70. Liu, Y., and Zhao, C. (2021, January 9–11). Research on FPGA-based Generative Adversarial Network implementation method. Proceedings of the 33rd China Simulation Conference, Harbin, China.
71. Vanhoucke, V., Senior, A., and Mao, M.Z. (2011, January 15). Improving the Speed of Neural Networks on CPUs. Proceedings of the Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, Granada, Spain.
72. Zhang, S., Cao, J., Zhang, Q., Zhang, Q., Zhang, Y., and Wang, Y. (2020, January 8–11). An fpga-based reconfigurable cnn accelerator for yolo. Proceedings of the 2020 IEEE 3rd International Conference on Electronics Technology (ICET), Chengdu, China.10.1109/ICET49382.2020.9119500 ↗
73. Li, Z., Chen, J., Wang, L., Cheng, B., Yu, J., and Jiang, S. (2020, January 11–14). CNN Weight Parameter Quantization Method for FPGA. Proceedings of the 2020 IEEE 6th International Conference on Computer and Communications (ICCC), Chengdu, China.10.1109/ICCC51575.2020.9345248 ↗
74. Chang, S.E., Li, Y., Sun, M., Shi, R., So, H.K.H., Qian, X., Wang, Y., and Lin, X. (March, January 27). Mix and match: A novel fpga-centric deep neural network quantization framework. Proceedings of the 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), Seoul, Korea.10.1109/HPCA51647.2021.00027 ↗
75. Zhao, X., Wang, Y., Cai, X., Liu, C., and Zhang, L. (2020, January 30). Linear Symmetric Quantization of Neural Networks for Low-Precision Integer Hardware. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
76. Bao, Z., Zhan, K., Zhang, W., and Guo, J. (2021, January 14–16). LSFQ: A Low Precision Full Integer Quantization for High-Performance FPGA-Based CNN Acceleration. Proceedings of the 2021 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS), Tokyo, Japan.10.1109/COOLCHIPS52128.2021.9410327 ↗
77. Research on Machine Learning Optimization Algorithm of CNN for FPGA Architecture10.1088/1742-6596/2006/1/012012 ↗
78. Design of FPGA recurrent neural network accelerator based on high level synthesis
79. Fowers, J., Ovtcharov, K., Strauss, K., Chung, E.S., and Sitt, G. (2014, January 11–13). A high memory bandwidth fpga accelerator for sparse matrix-vector multiplication. Proceedings of the 2014 IEEE 22nd Annual International Symposium on Field-Programmable Custom Computing Machines, Boston, MA, USA.10.1109/FCCM.2014.23 ↗
80. Nurvitadhi, E., Sheffield, D., Sim, J., Mishra, A., Venkatesh, G., and Marr, D. (2016, January 7–9). Accelerating binarized neural networks: Comparison of FPGA, CPU, GPU, and ASIC. Proceedings of the 2016 International Conference on Field-Programmable Technology (FPT), Xi’an, China.10.1109/FPT.2016.7929192 ↗
81. Gupta, A., and Suneja, K. (2020, January 13–15). Hardware Design of Approximate Matrix Multiplier based on FPGA in Verilog. Proceedings of the 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India.10.1109/ICICCS48265.2020.9121004 ↗
82. FPGA architecture for fast parallel computation of co-occurrence matrices10.1016/j.micpro.2006.02.013 ↗
83. Abbaszadeh, A., Iakymchuk, T., Bataller-Mompeán, M., Francés-Villora, J.V., and Rosado-Muñoz, A. (2019). Anscalable matrix computing unit architecture for FPGA, and SCUMO user design interface. Electronics, 8.10.3390/electronics8010094 ↗
84. Efficient cnn accelerator on fpga10.1080/03772063.2020.1821797 ↗
85. Kang, S., Lee, S., Kim, B., Kim, H., Sohn, K., Kim, N.S., and Lee, E. (March, January 27). An FPGA-based RNN-T Inference Accelerator with PIM-HBM. Proceedings of the 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Virtual.10.1145/3490422.3502355 ↗
86. Lavin, A., and Gray, S. (2016, January 27–30). Fast algorithms for convolutional neural networks. Proceedings of the IEEE conference on computer vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.435 ↗
87. Lu, L., and Liang, Y. (2018, January 24–29). SpWA: An efficient sparse winograd convolutional neural networks accelerator on FPGAs. Proceedings of the 55th Annual Design Automation Conference, San Francisco, CA, USA.10.1145/3195970.3196120 ↗
88. Kala, S., Mathew, J., Jose, B.R., and Nalesh, S. (2019, January 5–9). UniWiG: Unified winograd-GEMM architecture for accelerating CNN on FPGAs. Proceedings of the 2019 32nd International Conference on VLSI Design and 2019 18th International Conference on Embedded Systems (VLSID), Delhi, India.10.1109/VLSID.2019.00055 ↗
89. A power-efficient optimizing framework fpga accelerator based on winograd for yolo10.1109/ACCESS.2020.2995330 ↗
90. Winonn: Optimizing fpga-based convolutional neural network accelerators using sparse winograd algorithm10.1109/TCAD.2020.3012323 ↗
91. Design and optimization of target detection accelerator based on Winograd algorithm
92. Tang, F., Zhang, W., Tian, X., Fan, X., and Cao, X. (2018). Optimization of Convolution Neural Network Algorithm Based on FPGA. ESTC 2017. Communications in Computer and Information Science, Springer.10.1007/978-981-13-1026-3_10 ↗
93. Yu, F., Cao, Y., and Tang, Y. (2020, January 3–6). Realization of Quantized Neural Network for Super-resolution on PYNQ. Proceedings of the 2020 IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), Fayetteville, AR, USA.10.1109/FCCM48280.2020.00066 ↗
94. Ye, T., Kuppannagari, S.R., Kannan, R., and Prasanna, V.K. (September, January 30). Performance Modeling and FPGA Acceleration of Homomorphic Encrypted Convolution. Proceedings of the 2021 31st International Conference on Field-Programmable Logic and Applications (FPL), Dresden, Germany.10.1109/FPL53798.2021.00027 ↗
95. Zhang, H., Jiang, J., Fu, Y., and Chang, Y.C. (2021, January 22–24). Yolov3-tiny Object Detection SoC Based on FPGA Platform. Proceedings of the 2021 6th International Conference on Integrated Circuits and Microsystems (ICICM), Nanjing, China.10.1109/ICICM54364.2021.9660358 ↗
96. Xiao, C., Shi, C., Xu, D., Lin, F., and Ning, K. (2022, January 16–19). SDST-Accelerating GEMM-based Convolution through Smart Data Stream Transformation. Proceedings of the 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid), Taormina, Italy.10.1109/CCGrid54584.2022.00049 ↗
97. Özkilbaç, B., Ozbek, I.Y., and Karacali, T. (2022, January 9–10). Real-Time Fixed-Point Hardware Accelerator of Convolutional Neural Network on FPGA Based. Proceedings of the 2022 5th International Conference on Computing and Informatics (ICCI), New Cairo, Egypt.10.1109/ICCI54321.2022.9756093 ↗
98. Throughput-optimized FPGA accelerator for deep convolutional neural networks10.1145/3079758 ↗
99. Dnnvm: End-to-end compiler leveraging heterogeneous optimizations on fpga-based cnn accelerators10.1109/TCAD.2019.2930577 ↗
100. FPGA parallel structure design of convolutional neural network (CNN) algorithm