A Hybrid Deep Learning and Knowledge Graph Approach for Intelligent Image Indexing and Retrieval
Source abstract
Abstract
Technological advancements have enabled users to digitize and store an unlimited number of multimedia documents, including images and videos. However, the heterogeneous nature of multimedia content poses significant challenges in efficient indexing and retrieval. Traditional approaches primarily focus on visual features, often neglecting the semantic context, which limits retrieval efficiency. This paper proposes a hybrid deep learning and knowledge graph approach for intelligent image indexing and retrieval. By integrating deep learning models such as EfficientNet and Vision Transformer (ViT) with structured knowledge graphs, the proposed framework enhances semantic understanding and retrieval performance. The methodology incorporates feature extraction, concept classification, and hierarchical knowledge graph structuring to facilitate effective multimedia retrieval. Experimental results on benchmark datasets, including TRECVID, Corel, and MSCOCO, demonstrate significant improvements in precision, robustness, and query expansion techniques. The findings highlight the potential of combining deep learning with knowledge graphs to bridge the semantic gap and optimize multimedia indexing and retrieval.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, XLIM, UMR CNRS 7252, University of Limoges, Avenue Albert Thomas, 87060 Limoges, France
3iL Ingénieurs, 43 Rue de Sainte Anne, 87015 Limoges, France
References (100+)
Show source-provided references
1. Hamroun, M., Lajmi, S., Nicolas, H., and Amous, I. (2019, January 10–12). VISEN: A video interactive retrieval engine based on semantic network in large video collections. Proceedings of the 23rd International Database Applications & Engineering Symposium (IDEAS), New York, NY, USA.10.1145/3331076.3331094 ↗
2. Chen, J., Mao, J., Liu, Y., Zhang, F., Min, Z., and Ma, S. (2021, January 19–23). Towards a better understanding of query reformulation behavior in web search. Proceedings of the WWW ’21: The Web Conference 2021, Ljubljana, Slovenia.10.1145/3442381.3450127 ↗
3. Ntirogiannis, K., Gatos, B., and Pratikakis, I. (2011, January 18–21). Binarization of textual content in video frames. Proceedings of the 2011 International Conference on Document Analysis and Recognition (ICDAR), Beijing, China.10.1109/ICDAR.2011.141 ↗
4. Leow, W.K., Lew, M.S., Chua, T.S., Ma, W.Y., Chaisorn, L., and Bakker, E.M. (2005). The use and utility of high-level semantic features in video retrieval. Image and Video Retrieval, Springer.10.1007/11526346 ↗
5. A learned lexicon-driven paradigm for interactive video retrieval10.1109/TMM.2006.886275 ↗
6. Mediamill: Advanced browsing in news video archives10.1007/11788034_62 ↗
7. Vrochidis, S., Moumtzidou, A., King, P., Dimou, A., Mezaris, V., and Kompatsiaris, I. (2010, January 23–25). VERGE: A video interactive retrieval engine. Proceedings of the 2010 International Workshop on Content Based Multimedia Indexing (CBMI), Grenoble, France.10.1109/CBMI.2010.5529884 ↗
8. The vocabulary problem in human-system communication10.1145/32206.32212 ↗
9. On relevance, probabilistic indexing and information retrieval10.1145/321033.321035 ↗
10. Salton, G. (1971). Relevance Feedback in Information Retrieval. The SMART Retrieval System: Experiments in Automatic Document Processing, Prentice-Hall.
11. Jones, K.S. (2025, September 04). Automatic Keyword Classification for Information Retrieval. Available online: https://api.semanticscholar.org/CorpusID:62724133.
12. A theoretical basis for the use of co-occurrence data in information retrieval10.1108/eb026637 ↗
13. A non-classical logic for information retrieval10.1093/comjnl/29.6.481 ↗
14. Implementing a probabilistic information retrieval system
15. Yu, C.T., Buckley, C., Lam, K., and Salton, G. (1983). A Generalized Term Dependence Model in Information Retrieval, Cornell University. Technical Report.
16. Harman, D. (1992). Relevance Feedback Revisited, Association for Computing Machinery.10.1145/133160.133167 ↗
17. (2025, September 04). Statista: Average Number of Search Terms for Online Search Queries in the United States as of January 2020. Available online: https://www.statista.com/statistics/269740/number-of-search-terms-in-internet-research-in-the-us/.
18. (2025, September 04). Keyword Discovery: Keyword: Query Size by Country. Available online: https://www.keyworddiscovery.com/keyword-stats.html.
19. Improving query expansion using pseudo-relevant web knowledge for information retrieval10.1016/j.patrec.2022.04.013 ↗
20. Query expansion techniques for information retrieval: A survey10.1016/j.ipm.2019.05.009 ↗
21. A survey on visual content-based video indexing and retrieval10.1109/TSMCC.2011.2109710 ↗
22. Etter, D. (2025, September 04). KB Video Retrieval at TRECVID 2011, Available online: https://www-nlpir.nist.gov/projects/tvpubs/tv11.papers/kbvr.pdf.
23. Ellouze, N., Lammari, N., Métais, E., and Ahmed, M.B. (2010). CITOM: Approche de construction incrémentale d’une Topic Map multilingue. Data Knowl. Eng.
24. Rossetto, L., Giangreco, I., Ta, C., and Schuldt, H. (2017, January 6). Multimodal video retrieval with the 2017 IMO-TION system. Proceedings of the ICMR ’17: International Conference on Multimedia Retrieval (ICMR), New York, NY, USA.10.1145/3078971.3079012 ↗
25. A systematic review on content-based video retrieval10.1016/j.engappai.2020.103557 ↗
26. Wu, S., Li, Y., Zhu, K., Zhang, G., Liang, Y., Ma, K., Xiao, C., Zhang, H., Yang, B., and Chen, W. (2024). SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval. arXiv.10.18653/v1/2024.findings-acl.746 ↗
27. Representation learning: A review and new perspectives10.1109/TPAMI.2013.50 ↗
28. Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., Kim, S., and Kakao Brain Large-Scale AI Studio (2025, September 04). Coyo-700m: Image-Text Pair Dataset. GitHub. Available online: https://github.com/kakaobrain/coyo-dataset.
29. Chen, W., Hu, H., Chen, X., Verga, P., and Cohen, W. (2022, January 7–11). Murag: Multimodal retrieval-augmented generator for open question answering over images and text. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates.10.18653/v1/2022.emnlp-main.375 ↗
30. ML-LMCL: Mutual learning and large-margin contrastive learning for improving ASR robustness in spoken language understanding
31. Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., and Yue, X. (2023). LLaMA-Adapter V2: Parameter-efficient visual instruction model. arXiv.
32. Goldsack, T., Zhang, Z., Lin, C., and Scarton, C. (2023). Domain-Driven and Discourse-Guided Scientific Summarisation. European Conference on Information Retrieval, Springer.10.1007/978-3-031-28244-7_23 ↗
33. New process to identify audio concepts based on binary classifiers encapsulation10.7763/IJCEE.2012.V4.546 ↗
34. Elleuch, N., Zarka, M., Feki, I., Ba, A., and Alimi, A. (2010, January 15–17). Regimvid at Trecvid2010: Semantic Indexing. Proceedings of the TRECVID 2010 Workshop, Gaithersburg, MD, USA.
35. A generic framework for semantic video indexing based on visual concepts/contexts detection10.1007/s11042-014-1955-9 ↗
36. Content-based image retrieval at the end of the early years10.1109/34.895972 ↗
37. Semantic-based video retrieval survey10.4236/jcc.2018.68003 ↗
38. Sjoberg, M., Viitaniemi, V., Koskela, M., and Laaksonen, J. (2025, September 04). PicSOM Experiments in TRECVID 2009. Available online: https://research.cs.aalto.fi/cbir/papers/trecvid2009.pdf.
39. Slimi, J., Mansouri, S., Ammar, A.B., and Alimi, A.M. (2013, January 15–17). Video exploration tool based on semantic network. Proceedings of the 10th Conference on Open Research Areas in Information Retrieval, OAIR ’13, Lisbon, Portugal.
40. Slimi, J., Ammar, A.B., and Alimi, A.M. (2013, January 17–19). Interactive Video Data Visualization System Based on Semantic Organization. Proceedings of the 2013 11th International Workshop on Content-Based Multimedia Indexing (CBMI), Veszprem, Hungary.10.1109/CBMI.2013.6576575 ↗
41. Amato, G., Bolettieri, P., Carrara, F., Falchi, F., Gennaro, C., Messina, N., Vadicamo, L., and Vairo, C. (2023). VISIONE at video browser showdown 2023. MultiMedia Modeling: 29th International Conference, MMM 2023, Bergen, Norway, Jan. 9–12, 2023, Proceedings, Part I, Springer International Publishing.10.1007/978-3-031-27077-2_48 ↗
42. Fang, H., Xiong, P., Xu, L., and Chen, Y. (2021). Clip2video: Mastering video-text retrieval via image clip. arXiv.
43. Messina, N., Stefanini, M., Cornia, M., Baraldi, L., Falchi, F., Amato, G., and Cucchiara, R. (2022). ALADIN: Distilling fine-grained alignment scores for efficient image-text matching and retrieval. arXiv.10.1145/3549555.3549576 ↗
44. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Online.
45. Amato, G., Bolettieri, P., Carrara, F., Debole, F., Falchi, F., Gennaro, C., Vadicamo, L., and Vairo, C. (2021). The VISIONE Video Search System: Exploiting Off-the-Shelf Text Search Engines for Large-Scale Video Retrieval. J. Imaging, 7.10.3390/jimaging7050076 ↗
46. Large-scale instance-level image retrieval
47. Carrara, F., Vadicamo, L., Gennaro, C., and Amato, G. (2022). Approximate nearest neighbor search on standard search engines. Similarity Search and Applications, Springer.10.1007/978-3-031-17849-8_17 ↗
48. Gurrin, C., Zhou, L., Healy, G., Jónsson, B.Þ., Dang-Nguyen, D.-T., Lokoć, J., Tran, M.-T., Hürst, W., Rossetto, L., and Schöffmann, K. (2022, January 27–30). Introduction to the fifth annual lifelog search challenge, LSC’22. Proceedings of the ICMR ′22: International Conference on Multimedia Retrieval (ICMR’22), Newark, NJ, USA.10.1145/3512527.3531439 ↗
49. Interactive video retrieval evaluation at a distance: Comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown10.1007/s13735-021-00225-2 ↗
50. On influential trends in interactive video retrieval: Video Browser Showdown 2015–201710.1109/TMM.2018.2830110 ↗
51. Lokoč, J., Vopálková, Z., Dokoupil, P., and Peška, L. (2023). Video search with CLIP and interactive text query reformulation. MultiMedia Modeling: 29th International Conference, MMM 2023, Bergen, Norway, 9–12 January 2023, Proceedings, Part I, Springer.10.1007/978-3-031-27077-2_50 ↗
52. Halima, B.H., Hamroun, M., Moussa, S.B., and Alimi, A.M. (2013). An interactive engine for multilingual video browsing using semantic content. arXiv.
53. Tian, Q., Sebe, N., Qi, G.J., Huet, B., Hong, R., and Liu, X. (2016). Faceted navigation for browsing large video collection. MultiMedia Modeling, Springer.10.1007/978-3-319-27674-8 ↗
54. Galanopoulos, D., Markatopoulou, F., Mezaris, V., and Patras, I. (2017, January 6–9). Concept language models and event-based concept number selection for zero-example event detection. Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval ICMR ’17, New York, NY, USA.10.1145/3078971.3079043 ↗
55. Semantic concept based video retrieval using convolutional neural network10.1007/s42452-019-1870-9 ↗
56. Colace, F., De Santo, M., Moscato, V., Picariello, A., Schreiber, F.A., and Tanca, L. (2015). Content-Based Multimedia Retrieval. Data Management in Pervasive Systems, Springer.10.1007/978-3-319-20062-0 ↗
57. Faudemay, P., and Seyrat, C. (1997, January 1–2). Intelligent delivery of personalised video programmes from a video database. Proceedings of the Database and Expert Systems Applications, 8th International Conference (DEXA ’97), Toulouse, France.
58. Semi-Supervised Heterogeneous Fusion for Multimedia Data Co-Clustering10.1109/TKDE.2013.47 ↗
59. Poria, S., Chaturvedi, I., Cambria, E., and Hussain, A. (2016, January 12–15). Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings of the IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.10.1109/ICDM.2016.0055 ↗
60. Visual-textual sentiment classification with bi-directional multi-level attention networks10.1016/j.knosys.2019.04.018 ↗
61. Sentiment analysis of social images via hierarchical deep fusion of content and links10.1016/j.asoc.2019.04.010 ↗
62. Image-text sentiment analysis via deep multimodal attentive fusion10.1016/j.knosys.2019.01.019 ↗
63. Sentiment analysis using deep learning architectures: A review10.1007/s10462-019-09794-5 ↗
64. Xu, N. (2017, January 22–24). Analyzing multimodal public sentiment based on hierarchical semantic attentional network. Proceedings of the IEEE International Conference on Intelligence and Security Informatics (ISI), Beijing, China.10.1109/ISI.2017.8004895 ↗
65. Predicting microblog sentiments via weakly supervised multimodal deep learning10.1109/TMM.2017.2757769 ↗
66. An image-text consistency driven multimodal sentiment analysis approach for social media10.1016/j.ipm.2019.102097 ↗
67. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification10.1109/TASLP.2019.2957872 ↗
68. LSTM-based multi-label video event detection10.1007/s11042-017-5532-x ↗
69. Textual context-aware dense captioning with diverse words10.1109/TMM.2023.3241517 ↗
70. Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., and Wang, L. (2022, January 18–24). Scaling up vision-language pretraining for image captioning. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01745 ↗
71. Region-object relation-aware dense captioning via transformer10.1109/TNNLS.2022.3152990 ↗
72. Rahmani Video Index Point Detection and Extraction Framework Using Custom YoloV4 Darknet Object Detection Model10.1109/ACCESS.2021.3118048 ↗
73. Riedl, M., and Biemann, C. (2012, January 9–11). TopicTiling: A text segmentation algorithm based on LDA. Proceedings of the ACL 2012 Student Research Workshop, Jeju Island, Republic of Korea.
74. Segmentation and organization of lecture video based on visual contents
75. Podlesnaya, A., and Podlesnyy, S. (2016, January 21–22). Deep learning based semantic video indexing and retrieval. Proceedings of the SAI Intelligent Systems Conference (IntelliSys) 2016, London, UK.10.1007/978-3-319-56991-8_27 ↗
76. A novel approach for video text detection and recognition based on a corner response feature map and transferred deep convolutional neural network10.1109/ACCESS.2018.2851942 ↗
77. Li, Z., Liu, X., and Zhang, S. (2016, January 1–3). Shot boundary detection based on multilevel difference of colour histograms. Proceedings of the 2016 First International Conference on Multimedia and Image Processing (ICMIP), Bandar Seri Begawan, Brunei.10.1109/ICMIP.2016.24 ↗
78. Xu, J., Song, L., and Xie, R. (2016, January 27–30). Shot boundary detection using convolutional neural networks. Proceedings of the 2016 Visual Communications and Image Processing (VCIP), Chengdu, China.10.1109/VCIP.2016.7805554 ↗
79. Learning Video Moment Retrieval Without a Single Annotated Video10.1109/TCSVT.2021.3075470 ↗
80. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., and Liu, W. (2015, January 7–12). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR.2015.7298594 ↗
81. Supervised Matrix Factorization Hashing for Cross-Modal Retrieval10.1109/TIP.2016.2564638 ↗
82. Supervised deep hashing for scalable face image retrieval10.1016/j.patcog.2017.03.028 ↗
83. Liong, V.E., Lu, J., Wang, G., Moulin, P., and Zhou, J. (2015, January 7–12). Deep hashing for compact binary codes learning. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298862 ↗
84. Li, W.-J., Wang, S., and Kang, W.-C. (2016, January 9–15). Feature learning based deep supervised hashing with pairwise labels. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA.
85. Matas, J., Sebe, N., and Welling, M. (2016). Learning to Hash With Binary Deep Neural Network. Lecture Notes in Computer Science, Springer.10.1007/978-3-319-41754-7 ↗
86. Faghri, F., Fleet, D.J., Kiros, J.R., and Fidler, S. (2018). VSE++: Improving visual-semantic embeddings with hard negatives. arXiv.
87. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014). Microsoft COCO: Common objects in context. European Conference on Computer Vision, Springer.
88. Deep Semantic Multimodal Hashing Network for Scalable Image-Text and Video-Text Retrievals10.1109/TNNLS.2020.2997020 ↗
89. Ding, G., Guo, Y., and Zhou, J. (2014, January 23–28). Collective Matrix Factorization Hashing for Multimodal Data. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.267 ↗
90. Zhou, J., Ding, G., and Guo, Y. (2014, January 6–11). Latent semantic sparse hashing for cross-modal similarity search. Proceedings of the SIGIR ’14: The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, Gold Coast, QLD, Australia.10.1145/2600428.2609610 ↗
91. Lin, Z., Ding, G., Hu, M., and Wang, J. (2015, January 7–12). Semantics-preserving hashing for cross-view retrieval. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299011 ↗
92. Multimodal similarity-preserving hashing10.1109/TPAMI.2013.225 ↗
93. Jiang, Q.-Y., and Li, W.-J. (2017, January 21–26). Deep cross-modal hashing. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.348 ↗
94. Kunal, B., Kaur, K., and Choudhary, C. (2023, January 3–5). A Machine learning model for content-based image retrieval. Proceedings of the 2023 2nd International Conference for Innovation in Technology (INOCON), Bangalore, India.10.1109/INOCON57975.2023.10101215 ↗
95. Texture features for browsing and retrieval of image data10.1109/34.531803 ↗
96. Unsupervised segmentation of color-texture regions in images and video10.1109/34.946985 ↗
97. Lung segmentation on HRCT and volumetric CT for diffuse interstitial lung disease using deep convolutional neural networks10.1007/s10278-019-00254-8 ↗
98. An official American Thoracic Society/European Respiratory Society statement: Update of the international multidisciplinary classification of the idiopathic interstitial pneumonias10.1164/rccm.201308-1483ST ↗
99. Kunal, P., Singh, P., and Hirani, N. (2022, January 7–9). A Cohesive relation between cybersecurity and information security. Proceedings of the 2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT), Bangalore, India.10.1109/GCAT55367.2022.9972023 ↗
100. Content-based image retrieval of chest CT with convolutional neural network for diffuse interstitial lung disease: Performance assessment in three major idiopathic interstitial pneumonias10.3348/kjr.2020.0603 ↗