Multi-Objective Evolutionary Neural Architecture Search with Weight-Sharing Supernet
Source abstract
Abstract
Deep neural networks have played a crucial role in the field of deep learning, achieving significant success in practical applications. The architecture of neural networks is key to their performance. In the past few years, these architectures have been manually designed by experts with rich domain knowledge. Additionally, the optimal neural network architecture can vary depending on specific tasks and data distributions. Neural Architecture Search (NAS) is a class of techniques aimed at automatically searching for and designing neural network architectures according to the given tasks and data. Specifically, evolutionary-computation-based NAS methods are known for their strong global search capability and have aroused widespread interest in recent years. Although evolutionary-computation-based NAS has achieved success in a wide range of research and applications, it still faces bottlenecks in training and evaluating a large number of individuals during optimization. In this study, we first devise a multi-objective evolutionary NAS framework based on a weight-sharing supernet to improve the search efficiency of traditional evolutionary-computation-based NAS. This framework combines the population optimization characteristic of evolutionary algorithms with the weight-sharing ideas in one-shot models. We then design a bi-population MOEA/D algorithm based on the proposed framework to effectively solve the NAS problem. By constructing two sub-populations with different optimization objectives, the algorithm can effectively explore network architectures of various sizes in complex search spaces. An inter-population communication mechanism further enhances the algorithm’s exploratory capability, enabling it to find network architectures with uniform distribution and high diversity. Finally, we conduct performance comparison experiments on image classification datasets of different scales and complexities. Experimental results demonstrate the effectiveness of the proposed multi-objective evolutionary NAS framework and the practicality and transferability of the introduced bi-population MOEA/D-based NAS method compared to existing state-of-the-art NAS methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Metaheuristic Optimization Algorithms Research · Advanced Multi-Objective Optimization Algorithms · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China
School of Computer and Information Engineering, Henan Normal University, Xinxiang 453007, China
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen 518110, China
Funding
Guangdong Regional Joint Funds for Basic and Applied Research · Guangdong Natural Science Funds for Distinguished Young Scholars · National Natural Science Foundation of China · TCL Young Scholars Program
References (53)
Show source-provided references
1. Imagenet classification with deep convolutional neural networks
2. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
3. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
4. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
5. Zhang, Y., Chan, W., and Jaitly, N. (2017, January 5–9). Very deep convolutional networks for end-to-end speech recognition. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953077 ↗
6. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv.
7. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
8. A survey on evolutionary neural architecture search10.1109/TNNLS.2021.3100554 ↗
9. Neural architecture search: A survey
10. Lu, Z., Whalen, I., Boddeti, V., Dhebar, Y., Deb, K., Goodman, E., and Banzhaf, W. (2019, January 13–17). Nsga-net: Neural architecture search using multi-objective genetic algorithm. Proceedings of the Genetic and Evolutionary Computation Conference, Prague, Czech Republic.10.1145/3321707.3321729 ↗
11. Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q.V., and Kurakin, A. (2017, January 6–11). Large-scale evolution of image classifiers. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
12. Real, E., Aggarwal, A., Huang, Y., and Le, Q.V. (February, January 27). Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
13. Elsken, T., Metzen, J.H., and Hutter, F. (2018). Efficient multi-objective neural architecture search via lamarckian evolution. arXiv.10.1007/978-3-030-05318-5_3 ↗
14. Yang, Z., Wang, Y., Chen, X., Shi, B., Xu, C., Xu, C., Tian, Q., and Xu, C. (2020, January 13–19). Cars: Continuous evolution for efficient neural architecture search. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00190 ↗
15. Zoph, B., and Le, Q.V. (2016). Neural architecture search with reinforcement learning. arXiv.
16. Baker, B., Gupta, O., Naik, N., and Raskar, R. (2016). Designing neural network architectures using reinforcement learning. arXiv.
17. Liu, Y., Jia, X., Tan, M., Vemulapalli, R., Zhu, Y., Green, B., and Wang, X. (2020, January 13–19). Search to distill: Pearls are everywhere but not the eyes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00756 ↗
18. Cai, H., Chen, T., Zhang, W., Yu, Y., and Wang, J. (2018, January 2–7). Efficient architecture search by network transformation. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11709 ↗
19. Cai, H., Zhu, L., and Han, S. (2018). Proxylessnas: Direct neural architecture search on target task and hardware. arXiv.
20. Ahmed, K., and Torresani, L. (2018, January 8–14). Maskconnect: Connectivity learning by gradient descent. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01228-1_22 ↗
21. Liu, H., Simonyan, K., and Yang, Y. (2018). Darts: Differentiable architecture search. arXiv.
22. Dong, X., and Yang, Y. (2019, January 15–20). Searching for a robust neural architecture in four gpu hours. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00186 ↗
23. A survey of swarm and evolutionary computing approaches for deep learning10.1007/s10462-019-09719-2 ↗
24. Xie, L., and Yuille, A. (2017, January 22–29). Genetic cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.154 ↗
25. DeepMaker: A multi-objective optimization framework for deep neural networks in embedded systems10.1016/j.micpro.2020.102989 ↗
26. Completely automated CNN architecture design based on blocks10.1109/TNNLS.2019.2919608 ↗
27. Javaheripi, M., Samragh, M., Javidi, T., and Koushanfar, F. (2020, January 8–12). GeneCAI: Gene tic evolution for acquiring c ompact AI. Proceedings of the 2020 Genetic and Evolutionary Computation Conference, Cancún, Mexico.10.1145/3377930.3390226 ↗
28. A fast and elitist multiobjective genetic algorithm: NSGA-II10.1109/4235.996017 ↗
29. Mitchell, M. (1998). An Introduction to Genetic Algorithms, MIT Press.
30. Simulated binary crossover for continuous search space
31. Evolving deep convolutional neural networks for image classification10.1109/TEVC.2019.2916183 ↗
32. Lorenzo, P.R., and Nalepa, J. (2018, January 15–19). Memetic evolution of deep neural networks. Proceedings of the Genetic and Evolutionary Computation Conference, Kyoto, Japan.10.1145/3205455.3205631 ↗
33. Multiobjective evolutionary design of deep convolutional neural networks for image classification10.1109/TEVC.2020.3024708 ↗
34. MOEA/D: A multiobjective evolutionary algorithm based on decomposition10.1109/TEVC.2007.892759 ↗
35. Zhang, C., Shao, H., and Li, Y. (2000, January 8–11). Particle swarm optimisation for evolving artificial neural network. Proceedings of the SMC 2000 Conference Proceedings, 2000 IEEE International Conference on Systems, Man and Cybernetics, ‘Cybernetics Evolving to Systems, Humans, Organizations, and Their Complex Interactions’ (cat. no. 0. IEEE), Nashville, TN, USA.
36. Evolving image classification architectures with enhanced particle swarm optimisation10.1109/ACCESS.2018.2880416 ↗
37. He, C., Ye, H., Shen, L., and Zhang, T. (2020, January 13–19). Milenas: Efficient neural architecture search via mixed-level reformulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01201 ↗
38. Brock, A., Lim, T., Ritchie, J.M., and Weston, N. (2017). Smash: One-shot model architecture search through hypernetworks. arXiv.
39. Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor10.1109/TEVC.2019.2924461 ↗
40. Lu, Z., Deb, K., Goodman, E., Banzhaf, W., and Boddeti, V.N. (2020, January 23–28). Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search. Proceedings of the Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part I 16.10.1007/978-3-030-58452-8_3 ↗
41. Rawal, A., and Miikkulainen, R. (2018). From nodes to networks: Evolving recurrent neural networks. arXiv.10.1016/B978-0-12-815480-9.00015-3 ↗
42. Črepinšek, M., Liu, S.H., Mernik, M., and Ravber, M. (2019). Long term memory assistance for evolutionary algorithms. Mathematics, 7.10.3390/math7111129 ↗
43. Pham, H., Guan, M., Zoph, B., Le, Q., and Dean, J. (2018, January 10–15). Efficient neural architecture search via parameters sharing. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
44. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2018, January 18–23). Learning transferable architectures for scalable image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00907 ↗
45. Krizhevsky, A., and Hinton, G. (2024, January 30). Learning Multiple Layers of Features from Tiny Images, Available online: http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf.
46. Zheng, H., An, K., and Ou, Z. (2021, January 19–22). Efficient neural architecture search for end-to-end speech recognition via straight-through gradients. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Online Conference.10.1109/SLT48900.2021.9383527 ↗
47. Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J. (2020, January 23–28). Single path one-shot neural architecture search with uniform sampling. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XVI 16.10.1007/978-3-030-58517-4_32 ↗
48. Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., and Hutter, F. (2019, January 9–15). Nas-bench-101: Towards reproducible neural architecture search. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
49. Dong, X., and Yang, Y. (2020). Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv.
50. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K. (2017). Hierarchical representations for efficient architecture search. arXiv.
51. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
52. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
53. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.