Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
Source abstract
Abstract
Deep neural networks (DNNs) have been quite successful in solving many complex learning problems. However, DNNs tend to have a large number of learning parameters, leading to a large memory and computation requirement. In this paper, we propose a model compression framework for efficient training and inference of deep neural networks on embedded systems. Our framework provides data structures and kernels for OpenCL-based parallel forward and backward computation in a compressed form. In particular, our method learns sparse representations of parameters using ℓ 1 -based sparse coding while training, storing them in compressed sparse matrices. Unlike the previous works, our method does not require a pre-trained model as an input and therefore can be more versatile for different application environments. Even though the use of ℓ 1 -based sparse coding for model compression is not new, we show that it can be far more effective than previously reported when we use proximal point algorithms and the technique of debiasing. Our experiments show that our method can produce minimal learning models suitable for small embedded devices.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science, Hanyang University ERICA, Ansan 15588, Korea
Computer Science and Engineering, Hanyang University ERICA, Ansan 15588, Korea
Funding
Hanyang University
References (66)
Show source-provided references
1. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25, Curran Associates, Inc.
2. Framewise phoneme classification with bidirectional LSTM and other neural network architectures10.1016/j.neunet.2005.06.042 ↗
3. Natural Language Processing (Almost) from Scratch
4. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
5. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
6. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going Deeper with Convolutions. Proceedings of the Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
7. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and Understanding Convolutional Networks. Proceedings of the Computer Vision (ECCV 2014), Zurich, Switzerland.10.1007/978-3-319-10590-1_53 ↗
8. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.
9. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (July, January 26). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
10. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4–9). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
11. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
12. Taigman, Y., Yang, M., Ranzato, M., and Wolf, L. (2014, January 23–28). DeepFace: Closing the Gap to Human-Level Performance in Face Verification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.220 ↗
13. Coates, A., Huval, B., Wang, T., Wu, D., Catanzaro, B., and Andrew, N. (2013, January 16–21). Deep learning with COTS HPC systems. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA.
14. Lawrence, S., Giles, C.L., and Tsoi, A.C. (1997, January 27–31). Lessons in Neural Network Training: Overfitting May be Harder than Expected. Proceedings of the Fourtheenth National Conference on Artificial Intelligence (AAAI’97), Providence, RI, USA.
15. Cloud Computing and Emerging IT Platforms: Vision, Hype, and Reality for Delivering Computing As the 5th Utility10.1016/j.future.2008.12.001 ↗
16. Jaderberg, M., Vedaldi, A., and Zisserman, A. (2014, January 1–5). Speeding up Convolutional Neural Networks with Low Rank Expansions. Proceedings of the British Machine Vision Conference, Nottingham, UK.10.5244/C.28.88 ↗
17. Denton, E., Zaremba, W., Bruna, J., LeCun, Y., and Fergus, R. (2014). Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. Advances in Neural Information Processing Systems 27, Curran Associates, Inc.
18. Ioannou, Y., Robertson, D.P., Shotton, J., Cipolla, R., and Criminisi, A. (2016, January 2–4). Training cnns with low-rank filters for efficient image classification. Proceedings of the International Conference on Learning Representations, San Juan, PR, USA.
19. Tai, C., Xiao, T., Wang, X., and E, W. (2016, January 2–4). Convolutional neural networks with low-rank regularization. Proceedings of the International Conference on Learning Representations, San Juan, PR, USA.
20. Hanson, S.J., and Pratt, L.Y. (1989). Comparing Biases for Minimal Network Construction with Back-Propagation. Advances in Neural Information Processing Systems 1, Morgan-Kaufmann.
21. LeCun, Y., Denker, J.S., and Solla, S.A. (1990). Optimal Brain Damage. Advances in Neural Information Processing Systems 2, Morgan-Kaufmann.
22. Hassibi, B., and Stork, D.G. (1993). Second order derivatives for network pruning: Optimal Brain Surgeon. Advances in Neural Information Processing Systems 5, Morgan-Kaufmann.
23. Han, S., Pool, J., Tran, J., and Dally, W.J. (2015). Learning Both Weights and Connections for Efficient Neural Networks. Advances in Neural Information Processing Systems 28, Curran Associates, Inc.
24. Han, S., Mao, H., and Dally, W.J. (2016, January 2–4). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. Proceedings of the International Conference on Learning Representations, San Juan, PR, USA.
25. Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. (2016, January 5–10). Learning Structured Sparsity in Deep Neural Networks. Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS’16), Barcelona, Spain.
26. Lebedev, V., and Lempitsky, V. (2016, January 27–30). Fast ConvNets Using Group-Wise Brain Damage. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.280 ↗
27. Shalev-Shwartz, S., and Tewari, A. (2009, January 14–18). Stochastic Methods for l1 Regularized Loss Minimization. Proceedings of the 26th International Conference on Machine Learning (ICML), Montreal, QC, Canada.
28. Duchi, J.C., and Singer, Y. (2009). Efficient Learning using Forward-Backward Splitting. Advances in Neural Information Processing Systems 22, Curran Associates, Inc.
29. Zhou, H., Alvarez, J.M., and Porikli, F. (2016). Less Is More: Towards Compact CNNs. Computer Vision (ECCV 2016), Springer.10.1007/978-3-319-46493-0_40 ↗
30. Proximal Algorithms10.1561/2400000003 ↗
31. Hinton, G. (2014). CSC321. Introduction to Neural Networks and Machine Learning, Toronto University. Lecture 6e.
32. Kingma, D.P., and Ba, J.L. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
33. Carreira-Perpiñán, M.Á., and Idelbayev, Y. (2018, January 18–22). Learning-Compression Algorithms for Neural Net Pruning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00890 ↗
34. Multiplier and gradient methods10.1007/BF00927673 ↗
35. Fletcher, R. (1969). A method for nonlinear constraints in minimization problems. Optimization, Academic Press.
36. Chen, C., Tung, F., Vedula, N., and Mori, G. (2018, January 8–14). Constraint-Aware Deep Neural Network Compression. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_25 ↗
37. Zoph, B., and Le, Q.V. (2016). Neural Architecture Search with Reinforcement Learning. arXiv.
38. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2017). Learning Transferable Architectures for Scalable Image Recognition. arXiv.10.1109/CVPR.2018.00907 ↗
39. He, Y., Lin, J., Liu, Z., Wang, H., Li, L.J., and Han, S. (2018, January 8–14). AMC: AutoML for Model Compression and Acceleration on Mobile Devices. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_48 ↗
40. Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges10.1109/MSP.2017.2765695 ↗
41. Lee, H., Battle, A., Raina, R., and Ng, A.Y. (2007). Efficient sparse coding algorithms. Advances in Neural Information Processing Systems 19, MIT Press.10.7551/mitpress/7503.003.0105 ↗
42. CoSaMP: Iterative signal recovery from incomplete and inaccurate samples10.1016/j.acha.2008.07.002 ↗
43. Dictionary Learning for Sparse Coding: Algorithms and Convergence Analysis10.1109/TPAMI.2015.2487966 ↗
44. Regression Shrinkage and Selection via the Lasso10.1111/j.2517-6161.1996.tb02080.x ↗
45. The Dantzig Selector: Statistical Estimation When P Is Much Larger Than N
46. Decoding by linear programming10.1109/TIT.2005.858979 ↗
47. Compressed sensing10.1109/TIT.2006.871582 ↗
48. Matrix Completion with Noise10.1109/JPROC.2009.2035722 ↗
49. Robust Stochastic Approximation Approach to Stochastic Programming10.1137/070704277 ↗
50. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
51. Lee, S., and Wright, S. (July, January 28). Manifold Identification of Dual Averaging Methods for Regularized Stochastic Online Learning. Proceedings of the 28th International Conference on Machine Learning (ICML), Bellevue, WA, USA.
52. Interior Gradient and Proximal Methods for Convex and Conic Optimization10.1137/S1052623403427823 ↗
53. Nitanda, A. (2014). Stochastic Proximal Gradient Descent with Acceleration Techniques. Advances in Neural Information Processing Systems 27, Curran Associates, Inc.
54. Nonasymptotic convergence of stochastic proximal point methods for constrained convex optimization
55. Rosasco, L., Villa, S., and Vu, B.C. (2014). Convergence of Stochastic Proximal Gradient Algorithm. arXiv.
56. Some methods of speeding up the convergence of iteration method10.1016/0041-5553(64)90137-5 ↗
57. Sparse reconstruction by separable approximation10.1109/TSP.2009.2016892 ↗
58. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems10.1109/JSTSP.2007.910281 ↗
59. De-noising by soft thresholding10.1109/18.382009 ↗
60. (2018, August 31). OpenCL-Caffe. Available online: https://github.com/amd/OpenCL-caffe.
61. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014). Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv.10.1145/2647868.2654889 ↗
62. (2018, August 31). ViennaCL. Available online: http://viennacl.sourceforge.net.
63. Bell, N., and Garland, M. (2008). Efficient Sparse Matrix-Vector Multiplication on CUDA, NVIDIA Corporation. NVIDIA Technical Report NVR-2008-004.
64. Bell, N., and Garland, M. (2009, January 14–20). Implementing Sparse Matrix-vector Multiplication on Throughput-oriented Processors. Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, Portland, OR, USA.10.1145/1654059.1654078 ↗
65. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report.
66. He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 11–18). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV ’15), Santiago, Chile.10.1109/ICCV.2015.123 ↗