Unveil Fundamental Graph Properties for Neural Architecture Search
Source abstract
Abstract
ABSTRACT Deep learning profoundly impacts various areas, such as face recognition and language translation. Owing to the increasingly high computational costs of training neural architectures, it is intractable to manually examine the performance of various neural architectures, promoting the area of Neural architecture search (NAS) that enables artificial intelligence (AI) automation. Despite the advance of NAS in automatically discovering the optimal neural networks, the fundamental understanding of the structure of neural architectures is limited. To fill the gap, we propose a method dubbed NASGraph that converts neural architectures to graphs whose graph properties determine their performances, enabling us to search for or design strong architectures. On various standard NAS benchmarks, NASGraph outperforms baseline NAS methods and requires remarkably fewer computation resources. The numerical results illuminate the relationship between neural architecture and its performance, complementing other approaches. Therefore, combining our approach with these approaches leads to performance improvement, but not combining the others. Our study offers a new perspective on network science for AI, potentially advancing various aspects of machine learning and uncovering the black box of convolutional neural networks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Graph Neural Networks · Advanced Neural Network Applications · Graph Theory and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science Rensselaer Polytechnic Institute Troy USA
IBM Research Yorktown Heights USA
Funding
National Science Foundation · IBM Research
References (86)
Show source-provided references
1. C.Yu J.Wang C.Peng C.Gao G.Yu andN.Sang “Bisenet: Bilateral Segmentation Network for Real‐Time Semantic Segmentation ” inProceedings of the European Conference on Computer Vision (ECCV)(2018) 325–341.10.1007/978-3-030-01261-8_20 ↗
2. P.Wang P.Chen Y.Yuan et al. “Understanding Convolution for Semantic Segmentation ” in2018 IEEE Winter Conference on Applications of Computer Vision (WACV)(IEEE 2018) 1451–1460.10.1109/WACV.2018.00163 ↗
3. R.Strudel R.Garcia I.Laptev andC.Schmid “Segmenter: Transformer for Semantic Segmentation ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2021) 7262–7272.10.1109/ICCV48922.2021.00717 ↗
4. A.Kirillov E.Mintun N.Ravi et al. “Segment Anything ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2023) 4015–4026.10.1109/ICCV51070.2023.00371 ↗
5. L.Ke M.Ye M.Danelljan et al. “Segment Anything in High Quality ”Advances in Neural Information Processing Systems36(2024).10.52202/075280-1303 ↗
6. T. B.Brown “Language Models Are Few‐Shot Learners ”arXiv Preprint arXiv:2005.14165(2020).
7. H.Touvron T.Lavril G.Izacard et al. “Llama: Open and Efficient Foundation Language Models ”arXiv Preprint arXiv:2302.13971(2023).
8. H.Touvron L.Martin K.Stone et al. “Llama 2: Open Foundation and Fine‐Tuned Chat Models ”arXiv Preprint arXiv:2307.09288(2023).
9. R.Anil A. M.Dai O.Firat et al. “Palm 2 Technical Report ”arXiv Preprint arXiv:2305.10403(2023).
10. S.Wu O.Irsoy S.Lu et al. “Bloomberggpt: A Large Language Model for Finance ”arXiv Preprint arXiv:2303.17564(2023).
11. R.Rombach A.Blattmann D.Lorenz P.Esser andB.Ommer “High‐Resolution Image Synthesis With Latent Diffusion Models ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2022) 10684–10695.10.1109/CVPR52688.2022.01042 ↗
12. Photorealistic Text‐to‐Image Diffusion Models With Deep Language Understanding
13. L.Zhang A.Rao andM.Agrawala “Adding Conditional Control to Text‐to‐Image Diffusion Models ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2023) 3836–3847.10.1109/ICCV51070.2023.00355 ↗
14. N.Ruiz Y.Li V.Jampani Y.Pritch M.Rubinstein andK.Aberman “Dreambooth: Fine Tuning Text‐to‐Image Diffusion Models for Subject‐Driven Generation ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2023) 22500–22510.10.1109/CVPR52729.2023.02155 ↗
15. Laion‐5b: An Open Large‐Scale Dataset for Training Next Generation Image‐Text Models
16. K.He X.Zhang S.Ren andJ.Sun “Deep Residual Learning for Image Recognition ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2016) 770–778.10.1109/CVPR.2016.90 ↗
17. B.Zoph V.Vasudevan J.Shlens andQ. V.Le “Learning Transferable Architectures for Scalable Image Recognition ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2018) 8697–8710.10.1109/CVPR.2018.00907 ↗
18. E.Real A.Aggarwal Y.Huang andQ. V.Le “Regularized Evolution for Image Classifier Architecture Search ” inProceedings of the AAAI Conference on Artificial Intelligence vol.33(2019) 4780–4789.10.1609/aaai.v33i01.33014780 ↗
19. M.TanandQ.Le “Efficientnet: Rethinking Model Scaling for Convolutional Neural Networks ” inInternational Conference on Machine Learning(PMLR 2019) 6105–6114.
20. L.Zhan J.Fan P.Ye andJ.Cao “A2s‐Nas: Asymmetric Spectral‐Spatial Neural Architecture Search for Hyperspectral Image Classification ” inICASSP 2023–2023 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP)(IEEE 2023) 1–5.10.1109/ICASSP49357.2023.10094640 ↗
21. Y.Chen T.Yang X.Zhang G.Meng X.Xiao andJ.Sun “Detnas: Backbone Search for Object Detection ”Advances in Neural Information Processing Systems32(2019).
22. G.Ghiasi T.‐Y.Lin andQ. V.Le “Nas‐Fpn: Learning Scalable Feature Pyramid Architecture for Object Detection ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2019) 7036–7045.10.1109/CVPR.2019.00720 ↗
23. C.Liu L.‐C.Chen F.Schroff et al. “Auto‐Deeplab: Hierarchical Neural Architecture Search for Semantic Image Segmentation ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2019) 82–92.10.1109/CVPR.2019.00017 ↗
24. X.Zhang H.Xu H.Mo et al. “Dcnas: Densely Connected Neural Architecture Search for Semantic Image Segmentation ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2021) 13956–13967.10.1109/CVPR46437.2021.01374 ↗
25. B.ZophandQ. V.Le “Neural Architecture Search With Reinforcement Learning ”arXiv Preprint arXiv:1611.01578(2016).
26. H.Pham M.Guan B.Zoph Q.Le andJ.Dean “Efficient Neural Architecture Search via Parameters Sharing ” inInternational Conference on Machine Learning(PMLR 2018) 4095–4104.
27. X.Chu B.Zhang andR.Xu “Fairnas: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2021) 12239–12248.10.1109/ICCV48922.2021.01202 ↗
28. C.Liu B.Zoph M.Neumann et al. “Progressive Neural Architecture Search ” inProceedings of the European Conference on Computer Vision (ECCV)(2018) 19–34.10.1007/978-3-030-01246-5_2 ↗
29. 10.1007/s11263-020-01396-x10.1007/s11263-020-01396-x ↗
30. H.Liu K.Simonyan andY.Yang “Darts: Differentiable Architecture Search ”arXiv Preprint arXiv:1806.09055(2018).
31. X.Chu X.Wang B.Zhang S.Lu X.Wei andJ.Yan “Darts‐: Robustly Stepping Out of Performance Collapse Without Indicators ”arXiv Preprint arXiv:2009.01027(2020).
32. Y.Xu L.Xie X.Zhang et al. “Pc‐Darts: Partial Channel Connections for Memory‐Efficient Architecture Search ”arXiv Preprint arXiv:1907.05737(2019).
33. X.Chu T.Zhou B.Zhang andJ.Li “Fair Darts: Eliminating Unfair Advantages in Differentiable Architecture Search ” inEuropean Conference on Computer Vision(Springer 2020) 465–480.10.1007/978-3-030-58555-6_28 ↗
34. P.Ye B.Li Y.Li T.Chen J.Fan andW.Ouyang “B‐Darts: Beta‐Decay Regularization for Differentiable Architecture Search ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2022) 10874–10883.10.1109/CVPR52688.2022.01060 ↗
35. X.Ning Y.Zheng T.Zhao Y.Wang andH.Yang “A Generic Graph‐Based Neural Architecture Encoding Scheme for Predictor‐Based Nas ” inEuropean Conference on Computer Vision(Springer 2020) 189–204.10.1007/978-3-030-58601-0_12 ↗
36. C.White W.Neiswanger andY.Savani “Bananas: Bayesian Optimization With Neural Architectures for Neural Architecture Search ” inProceedings of the AAAI Conference on Artificial Intelligence12 vol.35(2021) 10293–10301.10.1609/aaai.v35i12.17233 ↗
37. Bridging the Gap Between Sample‐Based and One‐Shot Neural Architecture Search With Bonas
38. H.Zhou M.Yang J.Wang andW.Pan “Bayesnas: A Bayesian Approach for Neural Architecture Search ” inInternational Conference on Machine Learning(PMLR 2019) 7603–7613.
39. J.Mellor J.Turner A.Storkey andE. J.Crowley “Neural Architecture Search Without Training ” inInternational Conference on Machine Learning(PMLR 2021) 7588–7598.
40. M. S.Abdelfattah A.Mehrotra Ł.Dudziak andN. D.Lane “Zero‐Cost Proxies for Lightweight Nas ”arXiv Preprint arXiv:2101.08134(2021).
41. G.Li Y.Yang K.Bhardwaj andR.Marculescu “Zico: Zero‐Shot Nas via Inverse Coefficient of Variation on Gradients ”arXiv Preprint arXiv:2301.11300(2023).
42. G. F.Montufar R.Pascanu K.Cho andY.Bengio “On the Number of Linear Regions of Deep Neural Networks ”Advances in Neural Information Processing Systems27(2014).
43. B.HaninandD.Rolnick “Complexity of Linear Regions in Deep Networks ” inInternational Conference on Machine Learning(PMLR 2019) 2596–2604.
44. T.Serra C.Tjandraatmadja andS.Ramalingam “Bounding and Counting Linear Regions of Deep Neural Networks ” inInternational Conference on Machine Learning(PMLR 2018) 4558–4566.
45. S. S.Du X.Zhai B.Poczos andA.Singh “Gradient Descent Provably Optimizes Over‐Parameterized Neural Networks ”arXiv Preprint arXiv:1810.02054(2018).
46. A.Jacot F.Gabriel andC.Hongler “Neural Tangent Kernel: Convergence and Generalization in Neural Networks ”Advances in Neural Information Processing Systems31(2018).
47. A.BiettiandJ.Mairal “On the Inductive Bias of Neural Tangent Kernels ”Advances in Neural Information Processing Systems32(2019).
48. 10.1145/300534810.1145/3005348 ↗
49. Z.Liu M.Sun T.Zhou G.Huang andT.Darrell “Rethinking the Value of Network Pruning ”arXiv Preprint arXiv:1810.05270(2018).
50. M.Shen P.Molchanov H.Yin andJ. M.Alvarez “When to Prune? A Policy Towards Early Structural Pruning ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2022) 12247–12256.10.1109/CVPR52688.2022.01193 ↗
51. G.Fang X.Ma M.Song M. B.Mi andX.Wang “Depgraph: Towards Any Structural Pruning ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2023) 16091–16101.10.1109/CVPR52729.2023.01544 ↗
52. Y.Chen G.Meng Q.Zhang et al. “Renas: Reinforced Evolutionary Neural Architecture Search ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2019) 4787–4796.10.1109/CVPR.2019.00492 ↗
53. 10.1109/TEVC.2021.306146610.1109/TEVC.2021.3061466 ↗
54. 10.1109/TEVC.2021.309944810.1109/TEVC.2021.3099448 ↗
55. 10.1109/TEVC.2022.322756210.1109/TEVC.2022.3227562 ↗
56. Z.Cai L.Chen P.Liu T.Ling andY.Lai “Eg‐Nas: Neural Architecture Search With Fast Evolutionary Exploration ” inProceedings of the AAAI Conference on Artificial Intelligence vol.38(2024) 11159–11167.10.1609/aaai.v38i10.28993 ↗
57. J.You J.Leskovec K.He andS.Xie “Graph Structure of Neural Networks ” inInternational Conference on Machine Learning(PMLR 2020) 10881–10891.
58. S.Xie A.Kirillov R.Girshick andK.He “Exploring Randomly Wired Neural Networks for Image Recognition ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2019) 1284–1293.10.1109/ICCV.2019.00137 ↗
59. G.Kadlecová J.Lukasik M.Pilát et al. “Surprisingly Strong Performance Prediction With Neural Graph Features ”arXiv Preprint arXiv:2404.16551(2024).
60. 10.1038/s41467-024-48069-810.1038/s41467-024-48069-8 ↗
61. X.DongandY.Yang “Nas‐Bench‐201: Extending the Scope of Reproducible Neural Architecture Search ”arXiv Preprint arXiv:2001.00326(2020).
62. Pruning Neural Networks Without Any Data by Iteratively Conserving Synaptic Flow
63. J.FrankleandM.Carbin “The Lottery Ticket Hypothesis: Finding Sparse Trainable Neural Networks ”arXiv Preprint arXiv:1803.03635(2018).
64. I.Radosavovic J.Johnson S.Xie W.‐Y.Lo andP.Dollár “On Network Design Spaces for Visual Recognition ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2019) 1882–1890.10.1109/ICCV.2019.00197 ↗
65. Y.Duan X.Chen H.Xu et al. “Transnas‐Bench‐101: Improving Transferability and Generalizability of Cross‐Task Neural Architecture Search ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2021) 5251–5260.10.1109/CVPR46437.2021.00521 ↗
66. A.Krishnakumar C.White A.Zela R.Tu M.Safari andF.Hutter “Nas‐Bench‐Suite‐Zero: Accelerating Research on Zero Cost Proxies ”arXiv Preprint arXiv:2210.03230(2022).
67. C.White M.Safari R.Sukthanker et al. “Neural Architecture Search: Insights From 1000 Papers ”arXiv Preprint arXiv:2301.08727(2023).
68. X.Chen L.Xie J.Wu andQ.Tian “Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation ” inProceedings of the IEEE/CVF International Conference on Computer Vision(2019) 1294–1303.10.1109/ICCV.2019.00138 ↗
69. W.Chen X.Gong andZ.Wang “Neural Architecture Search on Imagenet in Four Gpu Hours: A Theoretically Inspired Perspective ”arXiv Preprint arXiv:2102.11535(2021).
70. 10.1109/5.72679110.1109/5.726791 ↗
71. A.Krizhevsky I.Sutskever andG. E.Hinton “Imagenet Classification With Deep Convolutional Neural Networks ”Advances in Neural Information Processing Systems25(2012).
72. K.SimonyanandA.Zisserman “Very Deep Convolutional Networks for Large‐Scale Image Recognition ”arXiv Preprint arXiv:1409.1556(2014).
73. A.Vaswani “Attention Is All You Need ”Advances in Neural Information Processing Systems(2017).
74. N.Lee T.Ajanthan andP. H.Torr “Snip: Single‐Shot Network Pruning Based on Connection Sensitivity ”arXiv Preprint arXiv:1810.02340(2018).
75. C.Wang G.Zhang andR.Grosse “Picking Winning Tickets Before Training by Preserving Gradient Flow ”arXiv Preprint arXiv:2002.07376(2020).
76. L.Theis I.Korshunova A.Tejani andF.Huszár “Faster Gaze Prediction With Dense Networks and Fisher Pruning ”arXiv Preprint arXiv:1801.05787(2018).
77. D.Zhou X.Zhou W.Zhang et al. “Econas: Finding Proxies for Economical Neural Architecture Search ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(2020) 11396–11404.10.1109/CVPR42600.2020.01141 ↗
78. J.Deng W.Dong R.Socher L.‐J.Li K.Li andL.Fei‐Fei “Imagenet: A Large‐Scale Hierarchical Image Database ” in2009 IEEE Conference on Computer Vision and Pattern Recognition(IEEE 2009) 248–255.10.1109/CVPR.2009.5206848 ↗
79. 10.1038/nature1694810.1038/nature16948 ↗
80. R.Gupta T.Roughgarden andC.Seshadhri “Decompositions of Triangle‐Dense Graphs ” inProceedings of the 5th Conference on Innovations in Theoretical Computer Science(2014) 471–482.10.1145/2554797.2554840 ↗
81. A.Krizhevsky G.Hinton et al. Learning Multiple Layers of Features From Tiny Images Technical Report(Univeristy of Toronto 2009).
82. P.Chrabaszcz I.Loshchilov andF.Hutter arXiv PreprintarXiv:1707.08819(2017).
83. A. R.Zamir A.Sax W.Shen L. J.Guibas J.Malik andS.Savarese inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2018) 3712–3722.10.1109/CVPR.2018.00391 ↗
84. H.Pham M.Guan B.Zoph Q.Le andJ.Dean inInternational Conference on Machine Learning(PMLR 2018) 4095–4104.
85. C.Liu B.Zoph M.Neumann et al. inProceedings of the European Conference on Computer Vision(2018) 19–34.10.1007/978-3-030-01246-5_2 ↗
86. T.DeVriesandG. W.Taylor arXiv PreprintarXiv:1708.04552(2017).