Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks
Source abstract
Abstract
Neural networks have become a cornerstone of computer vision applications, with tasks ranging from image classification to object detection. However, challenges such as hyperparameter optimization (HPO) and model compression remain critical for improving performance and deploying models on resource-constrained devices. In this work, we address these challenges using Tensor Network-based methods. For HPO, we propose and evaluate the TetraOpt algorithm against various optimization algorithms. These evaluations were conducted on subsets of the NATS-Bench dataset, including CIFAR-10, CIFAR-100, and ImageNet subsets. TetraOpt consistently demonstrated superior performance, effectively exploring the global optimization space and identifying configurations with higher accuracies. For model compression, we introduce a novel iterative method that combines CP, SVD, and Tucker tensor decompositions. Applied to ResNet-18 and ResNet-152, we evaluated our method on the CIFAR-10 and Tiny ImageNet datasets. Our method achieved compression ratios of up to 14.5× for ResNet-18 and 2.5× for ResNet-152. Additionally, the inference time for processing an image on a CPU remained largely unaffected, demonstrating the practicality of the method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Tensor decomposition and applications · Parallel Computing and Optimization Techniques · Model Reduction and Neural Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Terra Quantum AG, Kornhausstrasse 25, 9000 St. Gallen, Switzerland
References (41)
Show source-provided references
1. Hybrid quantum ResNet for car classification and its hyperparameter optimization10.1007/s42484-023-00123-2 ↗
2. Goreinov, S.A., Oseledets, I.V., Savostyanov, D.V., Tyrtyshnikov, E.E., and Zamarashkin, N.L. (2010). How to find a good submatrix. Matrix Methods: Theory, Algorithms and Applications: Dedicated to the Memory of Gene Golub, World Scientific Publishing Co Pte Ltd.10.1142/9789812836021_0015 ↗
3. NATS-Bench: Benchmarking NAS algorithms for architecture topology and size
4. Gardner, J.R., Kusner, M.J., Xu, Z.E., Weinberger, K.Q., and Cunningham, J.P. (2014, January 21–26). Bayesian optimization with inequality constraints. Proceedings of the International Conference on Machine Learning (ICML), Beijing, China.
5. Particle swarm optimization10.1109/ICNN.1995.488968 ↗
6. Differential evolution—A simple and efficient heuristic for global optimization over continuous spaces10.1023/A:1008202821328 ↗
7. DEAP: Evolutionary Algorithms Made Easy
8. Random search for hyper-parameter optimization
9. Tensor decompositions and applications10.1137/07070111X ↗
10. Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I., and Lempitsky, V. (2014). Speeding-up convolutional neural networks using fine-tuned CP-decomposition. arXiv.
11. Some mathematical notes on three-mode factor analysis10.1007/BF02289464 ↗
12. Golub, G.H., and Van Loan, C.F. (2013). Matrix Computations, JHU Press. [4th ed.].10.56021/9781421407944 ↗
13. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
14. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images in Technical Report TR-2009, University of Toronto.
15. Tiny ImageNet Visual Recognition Challenge
16. Snoek, J., Larochelle, H., and Adams, R.P. (2012). Practical Bayesian optimization of machine learning algorithms. Adv. Neural Inf. Process. Syst., 25.
17. Awad, N., Shala, G., Deng, D., Mallik, N., Feurer, M., Eggensperger, K., Biedenkapp, A., Vermetten, D., Wang, H., and Doerr, C. (2020). Squirrel: A switching hyperparameter optimizer. arXiv.
18. Bäck, T. (1996). Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms, Oxford University Press.10.1093/oso/9780195099713.001.0001 ↗
19. Vermetten, D., Wang, H., Doerr, C., and Bäck, T. (2019). Sequential vs. integrated algorithm selection and configuration: A case study for the modular CMA-ES. arXiv.
20. Hyperband: A novel bandit-based approach to hyperparameter optimization
21. Warmstarting of model-based algorithm configuration
22. Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2011, January 17–21). Sequential model-based optimization for general algorithm configuration. Proceedings of the Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy. Selected Papers.
23. Neural architecture search: A survey
24. Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automated Machine Learning: Methods, Systems, Challenges, Springer Nature.10.1007/978-3-030-05318-5 ↗
25. Zhang, T., Ye, S., Zhang, K., Tang, J., Wen, W., Fardad, M., and Wang, Y. (2018, January 8–14). A systematic DNN weight pruning framework using alternating direction method of multipliers. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_12 ↗
26. Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., and Shao, L. (2020, January 13–19). HRank: Filter pruning using high-rank feature map. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00160 ↗
27. Luo, J.-H., Wu, J., and Lin, W. (2017, January 22–29). ThiNet: A filter level pruning method for deep neural network compression. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.541 ↗
28. Discrimination-aware channel pruning for deep neural networks
29. Regularization-based pruning of irrelevant weights in deep neural architectures10.1007/s10489-022-04353-y ↗
30. Mitsuno, K., and Kurita, T. (2021, January 10–15). Filter pruning using hierarchical group sparse regularization for deep convolutional neural networks. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9413113 ↗
31. Network pruning using sparse learning and genetic algorithm10.1016/j.neucom.2020.03.082 ↗
32. An efficient unstructured sparse convolutional neural network accelerator for wearable ECG classification device
33. Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. (2016). Learning structured sparsity in deep neural networks. Adv. Neural Inf. Process. Syst., 29.
34. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2018, January 18–23). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00286 ↗
35. Novikov, A., Podoprikhin, D., Osokin, A., and Vetrov, D.P. (2015). Tensorizing neural networks. Adv. Neural Inf. Process. Syst., 28.
36. Nonlinear tensor train format for deep neural network compression10.1016/j.neunet.2021.08.028 ↗
37. Pan, Y., Xu, J., Wang, M., Ye, J., Wang, F., Bai, K., and Xu, Z. (February, January 27). Compressing recurrent neural networks with tensor ring for action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
38. Phan, A.-H., Sobolev, K., Sozykin, K., Ermilov, D., Gusak, J., Tichavský, P., Glukhov, V., Oseledets, I., and Cichocki, A. (2020, January 23–28). Stable low-rank tensor decomposition for compression of convolutional neural network. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XXIX.10.1007/978-3-030-58526-6_31 ↗
39. TT-cross approximation for multidimensional arrays10.1016/j.laa.2009.07.024 ↗
40. Introduction to evolutionary algorithms
41. ImageNet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗