From Classical Machine Learning to Deep Neural Networks: A Simplified Scientometric Review
Source abstract
Abstract
There are promising prospects on the way to widespread use of AI, as well as problems that need to be overcome to adapt AI&ML technologies in industries. The paper systematizes the AI sections and calculates the dynamics of changes in the number of scientific articles in machine learning sections according to Google Scholar. The method of data acquisition and calculation of dynamic indicators of changes in publication activity is described: growth rate (D1) and acceleration of growth (D2) of scientific publications. Analysis of publication activity, in particular, showed a high interest in modern transformer models, the development of datasets for some industries, and a sharp increase in interest in methods of explainable machine learning. Relatively small research domains are receiving increasing attention, as evidenced by the negative correlation between the number of articles and D1 and D2 scores. The results show that, despite the limitations of the method, it is possible to (1) identify fast-growing areas of research regardless of the number of articles, and (2) predict publication activity in the short term with satisfactory accuracy for practice (the average prediction error for the year ahead is 6%, with a standard deviation of 7%). This paper presents results for more than 400 search queries related to classified research areas and the application of machine learning models to industries. The proposed method evaluates the dynamics of growth and the decline of scientific domains associated with certain key terms. It does not require access to large bibliometric archives and allows to relatively quickly obtain quantitative estimates of dynamic indicators.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Economic and Technological Systems Analysis · Big Data and Business Intelligence · Statistical and Computational Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Satbayev University (KazNRTU), Satpayev str., 22A, Almaty 050013, Kazakhstan
Institute of Information and Computational Technologies MES RK, Pushkin str., 125, Almaty 050010, Kazakhstan
Funding
Science Committee of the Ministry of Education and Science of the Republic of Kazakhstan
References (100+)
Show source-provided references
1. (2021, May 10). The Socio-Economic Impact of AI in Healthcare. Available online: https://www.medtecheurope.org/wp-content/uploads/2020/10/mte-ai_impact-in-healthcare_oct2020_report.pdf.
2. Economic impact of artificial intelligence: New look for the macroeconomic assessment in Asia-Pacific region10.2991/ijcis.d.191025.001 ↗
3. Van Roy, V. (2020). AI Watch-National Strategies on Artificial Intelligence: A European Perspective in 2019, Joint Research Centre (Seville Site).
4. Citation analysis as a tool in journal evaluation10.1126/science.178.4060.471 ↗
5. The use of bibliometric analysis in research performance assessment and monitoring of interdisciplinary scientific developments
6. The measurement of Italian universities’ research productivity by a non parametric-bibliometric methodology10.1007/s11192-007-1942-2 ↗
7. Nanoscience and nanotechnologies at the Moscow domain: A bibliometric analysis based on Web of Science (Thomson Reuters)
8. Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key10.1023/B:SCIE.0000018532.70146.02 ↗
9. The effect of “open access” on citation impact: An analysis of ArXiv’s condensed matter section10.1002/asi.20663 ↗
10. Daim, T.U., Rueda, G.R., and Martin, H.T. (2005, January 31). Technology forecasting using bibliometric analysis and system dynamics. Proceedings of the A Unifying Discipline for Melting the Boundaries Technology Management, Portland, OR, USA.
11. Forecasting emerging technologies: Use of bibliometrics and patent analysis10.1016/j.techfore.2006.04.004 ↗
12. Inaba, T., and Squicciarini, M. (2017). ICT: A New Taxonomy Based on the International Patent Classification, Organisation for Economic Co-operation and Development.
13. Dynamic h-index: The Hirsch index in function of time10.1002/asi.20473 ↗
14. A proposal for a dynamic h-type index10.1002/asi.20890 ↗
15. Scale-independent indicators and research evaluation10.3152/147154300781782156 ↗
16. Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods10.1007/s11192-005-0008-6 ↗
17. New bibliometric indicators for prospectivity estimation of research fields
18. (2021, May 10). Artificial Intelligence. Available online: https://www.britannica.com/technology/artificial-intelligence.
19. Michael, M. (2021, May 10). Artificial Intelligence in Law: The State of Play 2016. Available online: https://www.neotalogic.com/wp-content/uploads/2016/04/Artificial-Intelligence-in-Law-The-State-of-Play-2016.pdf.
20. Machine learning and deep learning frameworks and libraries for large-scale data mining: A survey10.1007/s10462-018-09679-z ↗
21. Cruz, J.A., and Wishart, D.S. (2006). Applications of machine learning in cancer prediction and prognosis. Cancer Inform., 2.10.1177/117693510600200030 ↗
22. Deep learning for healthcare: Review, opportunities and challenges10.1093/bib/bbx044 ↗
23. A machine learning approach to predicting protein–ligand binding affinity with applications to molecular docking10.1093/bioinformatics/btq112 ↗
24. Machine learning for Internet of Things data analysis: A survey10.1016/j.dcan.2017.10.002 ↗
25. Farrar, C.R., and Worden, K. (2012). Structural Health Monitoring: A Machine Learning Perspective, John Wiley & Sons.10.1002/9781118443118 ↗
26. Lai, J., Qiu, J., Feng, Z., Chen, J., and Fan, H. (2016). Prediction of soil deformation in tunnelling using artificial neural networks. Comput. Intell. Neurosci., 2016.10.1155/2016/6708183 ↗
27. Liakos, K.G., Busato, P., Moshou, D., Pearson, S., and Bochtis, D. (2018). Machine learning in agriculture: A review. Sensors, 18.10.3390/s18082674 ↗
28. Applications of machine learning to ecological modelling10.1016/S0304-3800(01)00316-7 ↗
29. A systematic approach to geodynamic zoning based on artificial neural networks
30. Applications of machine learning to cognitive radio networks10.1109/MWC.2007.4300983 ↗
31. Data mining and machine learning in astronomy10.1142/S0218271810017160 ↗
32. Integration of results of recognition algorithms at the uranium deposits10.20965/jaciii.2014.p0347 ↗
33. Amirgaliev, E., Isabaev, Z., Iskakov, S., Kuchin, Y., Muhamediyev, R., Muhamedyeva, E., and Yakunin, K. (2014). Recognition of rocks at uranium deposits by using a few methods of machine learning. Soft Computing in Machine Learning, Springer.10.1007/978-3-319-05533-6_4 ↗
34. Application of one-class support vector machine to quickly identify multivariate anomalies from geochemical exploration data10.1144/geochem2016-024 ↗
35. Advances in natural language processing10.1126/science.aaa8685 ↗
36. A primer on neural network models for natural language processing10.1613/jair.4992 ↗
37. Types of machine learning algorithms
38. Taxonomy of machine learning algorithms to classify real time interactive applications
39. Machine learning methods: An overview
40. Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016). Deep Learning, MIT Press Cambridge.
41. Speech recognition using deep neural networks: A systematic review10.1109/ACCESS.2019.2896880 ↗
42. Hastie, T., Tibshirani, R., and Friedman, J. (2009). Unsupervised learning. The Elements of Statistical Learning, Springer.10.1007/978-0-387-84858-7 ↗
43. Supervised machine learning: A review of classification techniques
44. Data clustering: A review10.1145/331499.331504 ↗
45. Review of clustering algorithms. Non-standard parameter adaptation for exploratory data analysis
46. MacQueen, J. Some methods for classification and analysis of multivariate observations. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA. 27 December 1965–7 January 1966, 21 June–18 July 1965.
47. A global geometric framework for nonlinear dimensionality reduction10.1126/science.290.5500.2319 ↗
48. Nonlinear dimensionality reduction by locally linear embedding10.1126/science.290.5500.2323 ↗
49. Visualizing data using t-SNE
50. Schölkopf, B., Smola, A., and Müller, K.R. (1997). Kernel principal component analysis. International Conference on Artificial Neural Networks, Springer.10.7551/mitpress/1130.003.0026 ↗
51. Modern multidimensional scaling: Theory and applications10.1111/j.1745-3984.2003.tb01108.x ↗
52. An introduction to kernel and nearest-neighbor nonparametric regression10.1080/00031305.1992.10475879 ↗
53. The distance-weighted k-nearest-neighbor rule10.1109/TSMC.1976.5408784 ↗
54. (2021, May 10). K-nearest Neighbor Algorithm. Available online: http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm.
55. Support-vector networks10.1007/BF00994018 ↗
56. Neural networks for classification: A survey10.1109/5326.897072 ↗
57. (2021, May 10). The Neural Network Zoo. Available online: http://www.asimovinstitute.org/neural-network-zoo/.
58. Deep learning in neural networks: An overview10.1016/j.neunet.2014.09.003 ↗
59. Friedman, J.H. (2001). Greedy function approximation: A gradient boosting machine. Ann. Stat., 1189–1232.10.1214/aos/1013203451 ↗
60. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
61. (2021, May 10). Neural Network and Deep Learning. Available online: https://www.coursera.org/learn/neural-networks-deep-learning?specialization=deep-learning.
62. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
63. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv.
64. Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., and Coates, A. (2014). Deep speech: Scaling up end-to-end speech recognition. arXiv.
65. Jurafsky, D. (2000). Speech & Language Processing, Pearson Education India.
66. Liu, X. (2017). Deep recurrent neural network for protein function prediction from sequence. arXiv.10.1101/103994 ↗
67. Deep learning for sentiment analysis: A survey10.1002/widm.1253 ↗
68. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural architectures for named entity recognition. arXiv.10.18653/v1/N16-1030 ↗
69. Nayebi, A., and Vitelli, M. (2015). Gruv: Algorithmic music generation using recurrent neural networks. Course CS224D: Deep Learning for Natural Language Processing, Available online: https://cs224d.stanford.edu/reports/NayebiAran.pdf.
70. Lu, S., Zhu, Y., Zhang, W., Wang, J., and Yu, Y. (2018). Neural text generation: Past, present and beyond. arXiv.
71. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
72. Zelener, A. (2021, May 10). YAD2K: Yet Another Darknet 2 Keras. Available online: https://github.com/allanzelener/YAD2K.
73. Taigman, Y., Yang, M., Ranzato, M.A., and Wolf, L. (2014, January 23–28). Deepface: Closing the gap to human-level performance in face verification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.220 ↗
74. Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7–12). Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298682 ↗
75. Sankaranarayanan, S., Alavi, A., Castillo, C.D., and Chellappa, R. (2016, January 6–9). Triplet probabilistic embedding for face verification and clustering. Proceedings of the 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS), New York, NY, USA.10.1109/BTAS.2016.7791205 ↗
76. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
77. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
78. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial networks. arXiv.
79. (2021, May 09). Major Barriers to AI Adoption. Available online: https://www.agiloft.com/blog/barriers-to-ai-adoption/.
80. (2021, May 10). AI Adoption Advances, but Foundational Barriers Remain. Available online: https://www.mckinsey.com/featured-insights/artificial-intelligence/ai-adoption-advances-but-foundational-barriers-remain.
81. (2021, May 10). Machine Learning and the Five Vectors of Progress. Available online: https://www2.deloitte.com/us/en/insights/focus/signals-for-strategists/machine-learning-technology-five-vectors-of-progress.html.
82. The potential for artificial intelligence in healthcare10.7861/futurehosp.6-2-94 ↗
83. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
84. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why should i trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
85. Lundberg, S., and Lee, S.-I. (2017). A unified approach to interpreting model predictions. arXiv.
86. Muhamedyev, R.I., Yakunin, K.I., Kuchin, Y.A., Symagulov, A., Buldybayev, T., Murzakhmetov, S., and Abdurazakov, A. (2020). The use of machine learning “black boxes” explanation systems to improve the quality of school education. Cogent Eng., 7.10.1080/23311916.2020.1769349 ↗
87. (2021, May 10). ImageNet. Available online: http://image-net.org/index.
88. (2021, May 10). Open Images Dataset M5+ Extensions. Available online: https://storage.googleapis.com/openimages/web/index.html.
89. (2021, May 10). COCO Dataset. Available online: http://cocodataset.org/#home.
90. Synthetic dataset generation for object-to-model deep learning in industrial applications10.7717/peerj-cs.222 ↗
91. Ros, G., Sellart, L., Materzynska, J., Vazquez, D., and Lopez, A.M. (July, January 26). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
92. How to generate image dataset based on 3D model and deep learning method
93. Sim4cv: A photo-realistic simulator for computer vision applications10.1007/s11263-018-1073-7 ↗
94. Doan, A.-D., Jawaid, A.M., Do, T.-T., and Chin, T.-J. (2018). G2D: From GTA to Data. arXiv.
95. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017, January 13–15). CARLA: An open urban driving simulator. Proceedings of the Conference on Robot Learning, Mountain View, CA, USA.
96. One method of generating synthetic data to assess the upper limit of machine learning algorithms performance10.1080/23311916.2020.1718821 ↗
97. Arvanitis, T.N., White, S., Harrison, S., Chaplin, R., and Despotou, G. (2021). A method for machine learning generation of realistic synthetic datasets for Validating Healthcare Applications. medRxiv.10.1101/2021.02.11.21250741 ↗
98. Nikolenko, S.I. (2019). Synthetic data for deep learning. arXiv.
99. Mukhamedyev, R.I., Kuchin, Y., Denis, K., Murzakhmetov, S., Symagulov, A., and Yakunin, K. (2019, January 19–21). Assessment of the dynamics of publication activity in the field of natural language processing and deep learning. Proceedings of the International Conference on Digital Transformation and Global Society, Saint Petersburg, Russia.10.1007/978-3-030-37858-5_63 ↗
100. Latent dirichlet allocation