Assessing Efficiency in Artificial Neural Networks
Source abstract
Abstract
The purpose of this work was to develop an assessment technique and subsequent metrics that help in developing an understanding of the balance between network size and task performance in simple model networks. Here, exhaustive tests on simple model neural networks and datasets are used to validate both the assessment approach and the metrics derived from it. The concept of neural layer state space is introduced as a simple mechanism for understanding layer utilization, where a state is the on/off activation state of all neurons in a layer for an input. Neural efficiency is computed from state space to measure neural layer utilization, and a second metric called the artificial intelligence quotient (aIQ) was created to balance neural network performance and neural efficiency. To study aIQ and neural efficiency, two simple neural networks were trained on MNIST: a fully connected network (LeNet-300-100) and a convolutional neural network (LeNet-5). The LeNet-5 network with the highest aIQ was 2.32% less accurate but contained 30,912 times fewer parameters than the network with the highest accuracy. Both batch normalization and dropout layers were found to increase neural efficiency. Finally, networks with a high aIQ are shown to be resistant to memorization and overtraining as well as capable of learning proper digit classification with an accuracy of 92.51%, even when 75% of the class labels are randomized. These results demonstrate the utility of aIQ and neural efficiency as metrics for determining the performance and size of a small network using exemplar data.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Neural Networks and Applications · Adversarial Robustness in Machine Learning · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Center for Advancing Translational Science, National Institutes of Health, Rockville, MD 20850, USA
Axle Research and Technology, Rockville, MD 20850, USA
References (31)
Show source-provided references
1. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2019). ZeRO: Memory Optimization Towards Training A Trillion Parameter Models. arXiv.10.1109/SC41405.2020.00024 ↗
2. Du, N., Huang, Y., Dai, A.M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A.W., and Firat, O. (2022). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. arXiv.
3. Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., and Korthikanti, V. (2022). Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. arXiv.
4. Separability and geometry of object manifolds in deep neural networks10.1038/s41467-020-14578-5 ↗
5. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.10.1109/CVPR.2016.90 ↗
6. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv.
7. Hernández-García, A., and König, P. (2018). Lecture Notes in Computer Science, Springer.
8. Dropout: A Simple Way to Prevent Neural Networks from Overfitting
9. Antoniou, A., Storkey, A., and Edwards, H. (2017). Data Augmentation Generative Adversarial Networks. arXiv.
10. Sergeev, A., and Balso, M.D. (2018). Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv.
11. Intelligence and Changes in Regional Cerebral Glucose Metabolic Rate Following Learning10.1016/0160-2896(92)90018-M ↗
12. Cortical Glucose Metabolic Rate Correlates of Abstract Reasoning and Attention Studied with Positron Emission Tomography10.1016/0160-2896(88)90016-5 ↗
13. Regional Glucose Metabolic Changes after Learning a Complex Visuospatial/Motor Task: A Positron Emission Tomographic Study10.1016/0006-8993(92)90573-R ↗
14. Intelligence and Neural Efficiency10.1016/j.neubiorev.2009.04.001 ↗
15. Hirling, D., Tasnadi, E., Caicedo, J., Caroprese, M.V., Sjögren, R., Aubreville, M., Koos, K., and Horvath, P. (2023). Segmentation metric misinterpretations in bioimage analysis. Nat. Methods, 1–4.10.1038/s41592-023-01942-8 ↗
16. Buzhinsky, I., Nerinovsky, A., and Tripakis, S. (2021). Metrics and methods for robustness evaluation of neural networks with generative models. Mach. Learn., 1–36.10.1007/s10994-021-05994-9 ↗
17. Chen, J., Kao, S.H., He, H., Zhuo, W., Wen, S., Lee, C.H., and Chan, S.H.G. (2023). Run, Don’t Walk: Chasing Higher FLOPS for Faster Neural Networks. arXiv.10.1109/CVPR52729.2023.01157 ↗
18. Hernandez, D., and Brown, T.B. (2020). Measuring the Algorithmic Efficiency of Neural Networks. arXiv.
19. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
20. Touretzky, D.S. (1990). Advances in Neural Information Processing Systems 2, Morgan-Kaufmann.
21. Hanson, S.J., Cowan, J.D., and Giles, C.L. (1993). Advances in Neural Information Processing Systems 5, Morgan-Kaufmann.
22. Hu, H., Peng, R., Tai, Y.W., and Tang, C.K. (2016). Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures. arXiv.
23. Clevert, D.A., Unterthiner, T., and Hochreiter, S. (2015). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv.
24. Cohen, G., Afshar, S., Tapson, J., and van Schaik, A. (2017). EMNIST: An extension of MNIST to handwritten letters. arXiv.10.1109/IJCNN.2017.7966217 ↗
25. Lai, S.H., Lepetit, V., Nishino, K., and Sato, Y. (2016, January 11–14). Analysis on the Dropout Effect in Convolutional Neural Networks. Proceedings of the Computer Vision—ACCV 2016, Msterdam, The Netherlands.10.1007/978-3-319-54187-7 ↗
26. Labach, A., Salehinejad, H., and Valaee, S. (2019). Survey of Dropout Methods for Deep Neural Networks. arXiv.
27. Pereira, F., Burges, C.J.C., Bottou, L., and Weinberger, K.Q. (2012). Advances in Neural Information Processing Systems 25, Curran Associates, Inc.
28. Morcos, A.S., Barrett, D.G.T., Rabinowitz, N.C., and Botvinick, M. (2023, May 03). On the Importance of Single Directions for Generalization, Available online: http://xxx.lanl.gov/abs/1803.06959.
29. Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and Harnessing Adversarial Examples. arXiv.
30. Simple Models in Complex Worlds: Occam’s Razor and Statistical Learning Theory10.1007/s11023-022-09592-z ↗
31. Sun, K., and Nielsen, F. (2021). A Geometric Modeling of Occam’s Razor in Deep Learning. arXiv.