Metaheuristics Optimization with Deep Learning Enabled Automated Image Captioning System
Source abstract
Abstract
Image captioning is a popular topic in the domains of computer vision and natural language processing (NLP). Recent advancements in deep learning (DL) models have enabled the improvement of the overall performance of the image captioning approach. This study develops a metaheuristic optimization with a deep learning-enabled automated image captioning technique (MODLE-AICT). The proposed MODLE-AICT model focuses on the generation of effective captions to the input images by using two processes involving encoding unit and decoding unit. Initially, at the encoding part, the salp swarm algorithm (SSA), with a HybridNet model, is utilized to generate effectual input image representation using fixed-length vectors, showing the novelty of the work. Moreover, the decoding part includes a bidirectional gated recurrent unit (BiGRU) model used to generate descriptive sentences. The inclusion of an SSA-based hyperparameter optimizer helps in attaining effectual performance. For inspecting the enhanced performance of the MODLE-AICT model, a series of simulations were carried out, and the results are examined under several aspects. The experimental values suggested the betterment of the MODLE-AICT model over recent approaches.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, College of Sciences and Humanities-Aflaj, Prince Sattam Bin Abdulaziz University, Al-Kharj 16278, Saudi Arabia
Department of Information Technology, College of Computers and Information Technology, Taif University, P.O. Box 11099, Taif 21944, Saudi Arabia
Department of Informaion Systems, College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia
Department of Industrial Engineering, College of Engineering at Alqunfudah, Umm Al-Qura University, Mecca 24382, Saudi Arabia
Department of Computer Science, College of Science & Art at Mahayil, King Khalid University, Abha 62529, Saudi Arabia
Department of Computer Science, Faculty of Computers and Information, Menoufia University, Shibin El Kom 32511, Egypt
Department of Electrical Engineering and Computer Sciences, College of Engineering, University of California, Berkeley, CA 94720, USA
Department of Electrical Engineering, Faculty of Engineering & Technology, Future University in Egypt, New Cairo 11845, Egypt
Funding
King Khalid University · Princess Nourah bint Abdulrahman University · Scientific Research at Umm Al-Qura University
References (31)
Show source-provided references
1. A comprehensive survey of deep learning for image captioning10.1145/3295748 ↗
2. Sharma, H., Agrahari, M., Singh, S.K., Firoj, M., and Mishra, R.K. (2020, January 28–29). Image captioning: A comprehensive survey. In Proceedings of the 2020 International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC), Mathura, India.10.1109/PARC49193.2020.236619 ↗
3. Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., and Cucchiara, R. (2022). From show to tell: A survey on deep learning-based image captioning. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2022.3148210 ↗
4. Features to text: A comprehensive survey of deep learning on semantic segmentation and image captioning10.1155/2021/5538927 ↗
5. Revisiting image captioning via maximum discrepancy competition10.1016/j.patcog.2021.108358 ↗
6. Fuzzy cognitive maps with bird swarm intelligence optimization-based remote sensing image classification
7. Yao, T., Pan, Y., Li, Y., and Mei, T. (2018, January 8–14). Exploring visual relationship for image captioning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_42 ↗
8. Federated learning with blockchain assisted image classification for clustered UAV networks
9. Denoising-based multiscale feature fusion for remote sensing image captioning10.1109/LGRS.2020.2980933 ↗
10. Chohan, M., Khan, A., Mahar, M.S., Hassan, S., Ghafoor, A., and Khan, M. (2020). Image captioning using deep learning: A systematic literature review. Int. J. Adv. Comput. Sci. Appl., 11.10.14569/IJACSA.2020.0110537 ↗
11. Multi-level policy and reward-based deep reinforcement learning framework for image captioning10.1109/TMM.2019.2941820 ↗
12. Image captioning—A deep learning approach
13. High-resolution remote sensing image captioning based on structured attention10.1109/TGRS.2020.3042202 ↗
14. Toward remote sensing image retrieval under a deep image captioning perspective10.1109/JSTARS.2020.3013818 ↗
15. Image captioning with deep bidirectional LSTMs and multi-task learning
16. Chang, Y.H., Chen, Y.J., Huang, R.H., and Yu, Y.T. (2021). Enhanced Image Captioning with Color Recognition Using Deep Learning Methods. Appl. Sci., 12.10.3390/app12010209 ↗
17. Xiong, Y., Du, B., and Yan, P. (2019). Reinforced transformer for medical image captioning. International Workshop on Machine Learning in Medical Imaging, Springer.10.1007/978-3-030-32692-0_77 ↗
18. Improving image captioning with Pyramid Attention and SC-GAN10.1016/j.imavis.2021.104340 ↗
19. Image captioning model using attention and object features to mimic human image understanding
20. Multi-label semantic feature fusion for remote sensing image captioning10.1016/j.isprsjprs.2021.11.020 ↗
21. Robert, T., Thome, N., and Cord, M. (2018, January 8–14). Hybridnet: Classification and reconstruction cooperation for semi-supervised learning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_10 ↗
22. Salp Swarm Algorithm: A bio-inspired optimizer for engineering design problems10.1016/j.advengsoft.2017.07.002 ↗
23. Liu, J., Yang, Y., Lv, S., Wang, J., and Chen, H. (2019). Attention-based BiGRU-CNN for Chinese question classification. J. Ambient. Intell. Humaniz. Comput., 1–12.10.1007/s12652-019-01344-9 ↗
24. Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge10.1109/TPAMI.2016.2587640 ↗
25. Automatic image captioning based on ResNet50 and LSTM with soft attention. Wireless Communications and Mobile Computing10.1155/2020/8909458 ↗
26. Multilayer dense attention model for image caption10.1109/ACCESS.2019.2917771 ↗
27. Omri, M., Abdel-Khalek, S., Khalil, E.M., Bouslimi, J., and Joshi, G.P. (2022). Modeling of Hyperparameter Tuned Deep Learning Model for Automated Image Captioning. Mathematics, 10.10.3390/math10030288 ↗
28. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
29. Mnih, A., and Hinton, G. (2007, January 20–24). Three new graphical models for statistical language modelling. Proceedings of the ICML ’07.: 24th International Conference on Machine Learning, Corvalis, OR, USA.10.1145/1273496.1273577 ↗
30. Karpathy, A., and Li, F. (2015). Deep Visual-Semantic Alignments for Generating Image Descriptions, Stanford University.10.1109/CVPR.2015.7298932 ↗
31. Bujimalla, S., Subedar, M., and Tickoo, O. (2020). B-SCST: Bayesian self-critical sequence training for image captioning. arXiv.