A Modular Perspective on the Evolution of Deep Learning: Paradigm Shifts and Contributions to AI
Source abstract
Abstract
The rapid development of deep learning (DL) has demonstrated its modular contributions to artificial intelligence (AI) techniques, such as large language models (LLMs). DL variants have proliferated across domains such as feature extraction, normalization, lightweight architecture design, and module integration, yielding substantial advancements in these subfields. However, the absence of a unified review framework to contextualize DL’s modular evolutions within AI development complicates efforts to pinpoint future research directions. Existing review papers often focus on narrow technical aspects or lack systemic analysis of modular relationships, leaving gaps in our understanding how these innovations collectively drive AI progress. This work bridges this gap by providing a roadmap for researchers to navigate DL’s modular innovations, with a focus on balancing scalability and sustainability amid evolving AI paradigms. To address this, we systematically analyze extensive literature from databases including Web of Science, Scopus, arXiv, ACM Digital Library, IEEE Xplore, SpringerLink, Elsevier, etc., with the aim of (1) summarizing and updating recent developments in DL algorithms, with performance benchmarks on standard dataset; (2) identifying innovation trends in DL from a modular viewpoint; and (3) evaluating how these modular innovations contribute to broader advances in artificial intelligence, with particular attention to scalability and sustainability amid shifting AI paradigms.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Generative Adversarial Networks and Image Synthesis · Explainable Artificial Intelligence (XAI)
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Digital Economics and Management, Suzhou City University, Suzhou 215104, China
School of Statistics and Data Science, Southwestern University of Finance and Economics, Chengdu 611130, China
Faculty of Data Sciences, Shimonoseki City University, Yamaguchi 751-8510, Japan
Graduate School of Information, Production and Systems, Waseda University, Tokyo 169-8050, Japan
Funding
Suzhou City University Research Startup Funds
References (100+)
Show source-provided references
1. Computer science as empirical inquiry: Symbols and search10.1145/360018.360022 ↗
2. DENDRAL: A case study of the first expert system for scientific hypothesis formation10.1016/0004-3702(93)90068-M ↗
3. The perceptron: A probabilistic model for information storage and organization in the brain10.1037/h0042519 ↗
4. Neural networks and physical systems with emergent collective computational abilities10.1073/pnas.79.8.2554 ↗
5. Learning representations by back-propagating errors10.1038/323533a0 ↗
6. Deep Learning10.1038/nature14539 ↗
7. ImageNet classification with deep convolutional neural networks
8. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
9. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
10. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
11. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
12. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
13. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
14. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
15. Attention is all you need
16. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
17. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., and Sutskever, H. (2018). Improving Language Understanding by Generative Pre-Training, OpenAI. Technical Report.
18. Language models are unsupervised multitask learners
19. Language models are few-shot learners
20. Generative adversarial nets
21. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, PmLR, Vienna, Austria.
22. He, K., Chen, X., Xie, S., Li, Y., Dollar, P., and Girshick, R. (2022, January 18–24). Masked Autoencoders Are Scalable Vision Learners. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleands, LA, USA.10.1109/CVPR52688.2022.01553 ↗
23. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
24. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
25. Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv.
26. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (May, January 30). Graph Attention Networks. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
27. A survey on deep learning in medical image analysis10.1016/j.media.2017.07.005 ↗
28. Why the carbon footprint of generative large language models alone will not help us assess their sustainability10.1038/s42256-025-00979-y ↗
29. Interpretability research of deep learning: A literature survey10.1016/j.inffus.2024.102721 ↗
30. Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2020). Explainable ai: A review of machine learning interpretability methods. Entropy, 23.10.3390/e23010018 ↗
31. Addressing data dependency in neural networks: Introducing the Knowledge Enhanced Neural Network (KENN) for time series forecasting+10.1007/s10994-024-06714-9 ↗
32. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
33. Machine learning: Trends, perspectives, and prospects10.1126/science.aaa8415 ↗
34. Minar, M.R., and Naher, J. (2018). Recent advances in deep learning: An overview. arXiv.
35. A comprehensive survey on regularization strategies in machine learning10.1016/j.inffus.2021.11.005 ↗
36. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
37. On the importance of initialization and momentum in deep learning
38. A general survey on attention mechanisms in deep learning10.1109/TKDE.2021.3126456 ↗
39. A comprehensive survey on design and application of autoencoder in deep learning10.1016/j.asoc.2023.110176 ↗
40. Li, Z., Xia, T., Chang, Y., and Wu, Y. (2024). A Survey of Rwkv. arXiv.10.2139/ssrn.5111363 ↗
41. Chen, M., Mei, S., Fan, J., and Wang, M. (2024). An overview of diffusion models: Applications, guided generation, statistical rates and optimization. arXiv.
42. Ji, T., Hou, Y., and Zhang, D. (2024). A comprehensive survey on kolmogorov arnold networks (kan). arXiv.
43. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
44. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PmLR, Virutal.
45. Denoising Diffusion Probabilistic Models
46. Chen, X., Wang, L., and Zhang, H. (2023). Deep Learning in Medical Image Analysis: Current Trends and Future Directions. Information, 15.
47. A comparison review of transfer learning and self-supervised learning: Definitions, applications, advantages and limitations10.1016/j.eswa.2023.122807 ↗
48. Reinforcement learning, fast and slow10.1016/j.tics.2019.02.006 ↗
49. Neuroscience-Inspired Artificial Intelligence10.1016/j.neuron.2017.06.011 ↗
50. Methods for interpreting and understanding deep neural networks10.1016/j.dsp.2017.10.011 ↗
51. Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. (2017, January 21–26). Network dissection: Quantifying interpretability of deep visual representations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.354 ↗
52. Sabour, S., Frosst, N., and Hinton, G.E. (2017). Dynamic Routing Between Capsules. arXiv.
53. Inductive representation learning on large graphs
54. Graves, A., Wayne, G., and Danihelka, I. (2014). Neural Turing Machines. arXiv.
55. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A. (2008, January 5–9). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning, Hensenki, Finland.10.1145/1390156.1390294 ↗
56. Ng, A.Y. (2011). Sparse Autoencoder, Stanford University. CS294A Lecture Notes.
57. Zhang, X., Li, X., Wang, X., Wang, X., Wang, A., and Deng, J. (2021). Contrastive Learning of Visual Representations: A Survey. arXiv.
58. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
59. Tan, M., and Le, Q.V. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
60. Learning long-term dependencies with gradient descent is difficult10.1109/72.279181 ↗
61. Bidirectional recurrent neural networks10.1109/78.650093 ↗
62. Liu, S., Li, H., Shi, L., Ji, C., Cao, J., Lu, X., and Cao, Y. (2018). Independently recurrent neural network (IndRNN): Building a longer and deeper RNN. arXiv.10.1109/CVPR.2018.00572 ↗
63. Chen, Y., Zhang, Z., Yu, Y., Salakhutdinov, R., and Caruana, R. (2018, January 4–9). Dilated recurrent neural networks. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
64. Jaeger, H. (2001). Echo State Networks, GMD-Forschungszentrum Informationstechnik.
65. Bradbury, J., Merity, S., Xiong, C., Li, R., and Socher, R. (2016, January 2–4). Quasi-recurrent neural networks. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico.
66. Zhang, S.X., Zhao, R., Liu, C., Li, J., and Gong, Y. (2016, January 20–25). Recurrent support vector machines for speech recognition. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472806 ↗
67. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
68. xlstm: Extended long short-term memory
69. Alkin, B., Beck, M., Pöppel, K., Hochreiter, S., and Brandstetter, J. (2024). Vision-lstm: xlstm as generic vision backbone. arXiv.
70. Phased LSTM: Accelerating Recurrent Network Training for Long or Event-Sparse Time Series
71. Kalchbrenner, N., Danihelka, I., and Graves, A. (2015). Grid Long Short-Term Memory. arXiv.
72. Fukao, T., Iizuka, H., and Kurita, T. (2016). Multidimensional Long Short-Term Memory. arXiv.
73. Tai, K.S., Socher, R., and Manning, C.D. (2015, January 27–34). Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.10.3115/v1/P15-1150 ↗
74. Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., and Xu, B. (2016, January 22–27). Attention-based bidirectional long short-term memory networks for relation classification. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: SHORT Papers), Dublin, Ireland.10.18653/v1/P16-2034 ↗
75. Lei, T., Neubig, G., and Jaakkola, T. (2017). Training RNNs as Fast as CNNs. arXiv.
76. Tang, S., Li, B., and Yu, H. (2024). ChebNet: Efficient and stable constructions of deep neural networks with rectified power units via Chebyshev approximation. Commun. Math. Stat., 1–27.10.1007/s40304-023-00392-0 ↗
77. Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., Hou, T.Y., and Tegmark, M. (2024). Kan: Kolmogorov-arnold networks. arXiv.
78. Xu, K., Chen, L., and Wang, S. (2024). Kolmogorov-arnold networks for time series: Bridging predictive power and interpretability. arXiv.
79. Tang, T., Chen, Y., and Shu, H. (2024). 3D U-KAN implementation for multi-modal MRI brain tumor segmentation. arXiv.
80. Physics-informed kolmogorov-arnold networks for power system dynamics10.1109/OAJPE.2025.3529928 ↗
81. Gal, Y., and Ghahramani, Z. (2016, January 20–22). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning, PMLR, New York, NY, USA.
82. Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015). Weight uncertainty in neural networks. arXiv.
83. Fortunato, M., Blundell, C., and Vinyals, O. (2017). Bayesian recurrent neural networks. arXiv.
84. Physics-informed machine learning: A comprehensive review on applications in anomaly detection and condition monitoring10.1016/j.eswa.2024.124678 ↗
85. Lütjens, B., Crawford, C.H., Veillette, M., and Newman, D. (2021, January 14). Spectral pinns: Fast uncertainty propagation with physics-informed neural networks. Proceedings of the Symbiosis of Deep Learning and Differential Equations, Virutal.
86. A high-efficient hybrid physics-informed neural networks based on convolutional neural network10.1109/TNNLS.2021.3070878 ↗
87. Theory-guided hard constraint projection (HCP): A knowledge-based data-driven scientific machine learning method10.1016/j.jcp.2021.110624 ↗
88. Physics-informed machine learning10.1038/s42254-021-00314-5 ↗
89. Khalid, S., Yazdani, M.H., Azad, M.M., Elahi, M.U., Raouf, I., and Kim, H.S. (2024). Advancements in Physics-Informed Neural Networks for Laminated Composites: A Comprehensive Review. Mathematics, 13.10.3390/math13010017 ↗
90. Hasani, R., Lechner, M., Amini, A., Rus, D., and Grosu, R. (2021, January 2–9). Liquid Time-constant Networks. Proceedings of the AAAI Conference on Artificial Intelligence, Virutal.10.1609/aaai.v35i9.16936 ↗
91. Closed-form continuous-time neural networks10.1038/s42256-022-00556-7 ↗
92. Kumar, K., Verma, A., Gupta, N., and Yadav, A. (2023, January 23–24). Liquid Neural Networks: A Novel Approach to Dynamic Information Processing. Proceedings of the 2023 International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT), Faridabad, India.10.1109/ICAICCIT60255.2023.10466162 ↗
93. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
94. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
95. Karras, T., Laine, S., and Aila, T. (2019, January 15–20). A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗
96. Karras, T., Aittala, M., Laine, S., Lehtinen, J., and Aila, T. (2021, January 10–17). Alias-free generative adversarial networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.
97. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv.
98. CLIP-GAN: Stacking CLIPs and GAN for Efficient and Controllable Text-to-Image Synthesis10.1109/TMM.2025.3535304 ↗
99. Wu, J., Zhang, C., Xue, T., Freeman, W.T., and Tenenbaum, J.B. (2016, January 5–10). Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain.
100. Projected gans converge faster