DCAT: A Novel Transformer-Based Approach for Dynamic Context-Aware Image Captioning in the Tamil Language
Source abstract
Abstract
The task of image captioning in low-resource languages like Tamil is fraught with challenges due to limited linguistic resources and complex semantic structures. This paper addresses the problem of generating contextually and linguistically coherent captions in Tamil. We introduce the Dynamic Context-Aware Transformer (DCAT), a novel approach that synergizes the Vision Transformer (ViT) with the Generative Pre-trained Transformer (GPT-3), reinforced by a unique Context Embedding Layer. The DCAT model, tailored for Tamil, innovatively employs dynamic attention mechanisms during its Initialization, Training, and Inference phases to focus on pertinent visual and textual elements. Our method distinctively leverages the nuances of Tamil syntax and semantics, a novelty in the realm of low-resource language image captioning. Comparative evaluations against established models on datasets like Flickr8k, Flickr30k, and MSCOCO reveal DCAT’s superiority, with a notable 12% increase in BLEU score (0.7425) and a 15% enhancement in METEOR score (0.4391) over leading models. Despite its computational demands, DCAT sets a new benchmark for image captioning in Tamil, demonstrating potential applicability to other similar languages.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Technology, Karpagam College of Engineering, Myleripalayam Village, Coimbatore 641032, India
Department of Computer Science and Engineering, Karpagam College of Engineering, Myleripalayam Village, Coimbatore 641032, India
Department of Computer Science and Engineering, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, Chennai 600062, India
School of Computer Science and Engineering, VIT-AP University, Amaravati 522241, India
Power Electronics, Machines and Control (PEMC) Research Institute, University of Nottingham, 15 Triumph Rd., Lenton, Nottingham NG7 2GT, UK
Funding
National Research and Development Agency (ANID) through the FONDECYT Regular · National Council of Science and Technology (CONACYT) · British Council
References (36)
Show source-provided references
1. Mao, Y., Chen, L., Jiang, Z., Zhang, D., Zhang, Z., Shao, J., and Xiao, J. (2022, January 10–14). Rethinking the Reference-based Distinctive Image Captioning. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3548358 ↗
2. A Comprehensive Survey of Deep Learning for Image Captioning10.1145/3295748 ↗
3. Learning Transferable Perturbations for Image Captioning
4. Bottom-up and Top-down Object Inference Networks for Image Captioning10.1145/3580366 ↗
5. Arora, K., Raj, A., Goel, A., and Susan, S. (2022). A Hybrid Model for Combining Neural Image Caption and k-Nearest Neighbor Approach for Image Captioning. Soft Computing and Signal Processing, Springer.10.1007/978-981-16-1249-7_6 ↗
6. Oboyob: A sequential-semantic Bengali image captioning engine10.3233/JIFS-179351 ↗
7. Deep image captioning: A review of methods, trends and future challenges10.1016/j.neucom.2023.126287 ↗
8. GAGPT-2: A Geometric Attention-based GPT-2 Framework for Image Captioning in Hindi
9. V, J.P., and S, A.A.V. (ACM Trans. Asian Low-Resour. Lang. Inf. Process., 2023). Cross-Lingual Sentiment Analysis of Tamil Language Using a Multi-Stage Deep Learning Architecture, ACM Trans. Asian Low-Resour. Lang. Inf. Process., just accepted.10.1145/3631391 ↗
10. Sharma, H., Agrahari, M., Singh, S.K., Firoj, M., and Mishra, R.K. (2020, January 28–29). Image Captioning: A Comprehensive Survey. Proceedings of the International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC), Mathura, Uttar Pradesh, India.10.1109/PARC49193.2020.236619 ↗
11. A survey on deep neural network-based image captioning10.1007/s00371-018-1566-y ↗
12. Singh, P., Gupta, P., and Jain, H. (2022, January 28–30). A Comparative Study of Machine Learning Based Image Captioning Models. Proceedings of the 6th International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India.10.1109/ICOEI53556.2022.9777153 ↗
13. Image Captioning Encoder–Decoder Models Using CNN-RNN Architectures: A Comparative Study10.1007/s00034-022-02050-2 ↗
14. Yu, B., Zhong, Z., Qin, X., Yao, J., Wang, Y., and He, P. (2022, January 18–22). Automated testing of image captioning systems. Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual, Republic of Korea.10.1145/3533767.3534389 ↗
15. Lyu, Y., Zhao, Q., and Liu, Y. (2022, January 3–6). End-to-end image captioning based on reduced feature maps of deep learners pre-trained for object detection. Proceedings of the International Conference on Research in Adaptive and Convergent Systems, Virtual, Japan.10.1145/3538641.3561491 ↗
16. From Show to Tell: A Survey on Deep Learning-Based Image Captioning10.1109/TPAMI.2022.3148210 ↗
17. Cross modification attention-based deliberation model for image captioning
18. Transformer-based local-global guidance for image captioning10.1016/j.eswa.2023.119774 ↗
19. MAENet: A novel multi-head association attention enhancement network for completing intra-modal interaction in image captioning10.1016/j.neucom.2022.11.045 ↗
20. Zeng, P., Zhu, J., Song, J., and Gao, L. (2022, January 10–14). Progressive Tree-Structured Prototype Network for End-to-End Image Captioning. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3548024 ↗
21. Dynamic-balanced double-attention fusion for image captioning10.1016/j.engappai.2022.105194 ↗
22. Dynamic Convolution-based Encoder-Decoder Framework for Image Captioning in Hindi
23. When Pairs Meet Triplets: Improving Low-Resource Captioning via Multi-Objective Optimization
24. A Deep Ensemble Network for Sentiment Analysis in Bi-Lingual Low-Resource Languages10.1145/3600229 ↗
25. Development of an Efficient Method to Detect Mixed Social Media Data with Tamil-English Code Using Machine Learning Techniques10.1145/3563775 ↗
26. Tamil Offensive Language Detection: Supervised versus Unsupervised Learning Approaches10.1145/3575860 ↗
27. Kumar, V.H.V., and Lalithamani, N. (2022, January 17–19). English to Tamil Multi-Modal Image Captioning Translation. Proceedings of the IEEE World Conference on Applied Intelligence and Computing (AIC), Sonbhadra, India.
28. Gao, Y., Wang, N., Suo, W., Sun, M., and Wang, P. (2022, January 27–30). Improving Image Captioning via Enhancing Dual-Side Context Awareness. Proceedings of the 2022 International Conference on Multimedia Retrieval, Newark, NJ, USA.10.1145/3512527.3531379 ↗
29. Image captioning with adaptive incremental global context attention10.1007/s10489-021-02734-3 ↗
30. An Object Localization-based Dense Image Captioning Framework in Hindi
31. Rajendran, S., Kumar, M.A., Rajalakshmi, R., Dhanalakshmi, V., Balasubramanian, P., and Soman, K.P. (2023). Tamil NLP Technologies: Challenges, State of the Art, Trends and Future Scope. Speech and Language Technologies for Low-Resource Languages, Springer.10.1007/978-3-031-33231-9_6 ↗
32. A novel deep fuzzy neural network semantic-enhanced method for automatic image captioning10.1007/s00500-023-09100-0 ↗
33. Deep Semantic and Attentive Network for Unsupervised Video Summarization10.1145/3477538 ↗
34. Hong, I.P., Choi, G.H., Kim, P.K., and Choi, C. (2023, January 27–31). Security Verification Software Platform of Data-efficient Image Transformer Based on Fast Gradient Sign Method. Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, Tallinn, Estonia.10.1145/3555776.3577731 ↗
35. SAM: Cross-modal semantic alignments module for image-text retrieval10.1007/s11042-023-15798-9 ↗
36. Semantic Representations With Attention Networks for Boosting Image Captioning10.1109/ACCESS.2023.3268744 ↗