Graph-Propagated Multi-Scale Hashing with Contrastive Learning for Unsupervised Cross-Modal Retrieval
Source abstract
Abstract
This paper introduces Graph-Propagated Multi-Scale Hashing with Contrastive Learning (GPMCL), a novel unsupervised cross-modal hashing framework designed to address the semantic deficiency in large-scale unlabeled multimodal data. GPMCL first constructs an initial similarity matrix via cross-modal graph propagation, effectively capturing potential inter-modal relationships. A multi-scale enhancement strategy is then employed to integrate both local and global similarities, resulting in a more informative and robust similarity representation. To adaptively distinguish sample relationships, a Gaussian Mixture Model (GMM) is utilized to determine dynamic thresholds. Additionally, contrastive learning is incorporated in the feature space to enhance intra-class compactness and inter-class separability. Extensive experiments conducted on three public benchmark datasets demonstrate that GPMCL consistently outperforms existing state-of-the-art unsupervised cross-modal hashing methods in terms of retrieval performance. These results validate the effectiveness and generalization capability of the proposed method, highlighting its potential for practical cross-modal retrieval applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Advanced Neural Network Applications · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Electronics and Information Engineering, Shanghai University of Electric Power, Shanghai 200090, China
References (45)
Show source-provided references
1. Multimodal Machine Learning: A Survey and Taxonomy10.1109/TPAMI.2018.2798607 ↗
2. Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G.R.G., Levy, R., and Vasconcelos, N. (2010, January 25–29). A new approach to cross-modal multimedia retrieval. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.10.1145/1873951.1873987 ↗
3. Liu, H., Wang, R., Shan, S., and Chen, X. (2016, January 27–30). Deep Supervised Hashing for Fast Image Retrieval. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.227 ↗
4. A Review of Hashing Methods for Multimodal Retrieval10.1109/ACCESS.2020.2968154 ↗
5. Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions10.1109/JPROC.2024.3525147 ↗
6. Deep Adaptively-Enhanced Hashing With Discriminative Similarity Guidance for Unsupervised Cross-Modal Retrieval10.1109/TCSVT.2022.3172716 ↗
7. Unsupervised Cross-Modal Hashing via Semantic Text Mining10.1109/TMM.2023.3243608 ↗
8. Pseudo-label driven deep hashing for unsupervised cross-modal retrieval10.1007/s13042-023-01842-5 ↗
9. An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges10.1109/TCSVT.2017.2705068 ↗
10. Lin, Z., Ding, G., Hu, M., and Wang, J. (2015, January 7–12). Semantics-preserving hashing for cross-view retrieval. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299011 ↗
11. Supervised Matrix Factorization Hashing for Cross-Modal Retrieval10.1109/TIP.2016.2564638 ↗
12. Learning Discriminative Binary Codes for Large-scale Cross-modal Retrieval10.1109/TIP.2017.2676345 ↗
13. SCRATCH: A Scalable Discrete Matrix Factorization Hashing Framework for Cross-Modal Retrieval10.1109/TCSVT.2019.2911359 ↗
14. Jiang, Q.Y., and Li, W.J. (2017, January 21–26). Deep Cross-Modal Hashing. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.348 ↗
15. Tu, J., Liu, X., Lin, Z., Hong, R., and Wang, M. (2022, January 10–14). Differentiable Cross-modal Hashing via Multimodal Transformers. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3548187 ↗
16. Deep Semantic-Aware Proxy Hashing for Multi-Label Cross-Modal Retrieval10.1109/TCSVT.2023.3285266 ↗
17. Weiss, Y., Torralba, A., and Fergus, R. (2008, January 8–10). Spectral hashing. Proceedings of the 22nd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
18. Kumar, S., and Udupa, R. (2011, January 16–22). Learning hash functions for cross-view similarity search. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, Barcelona, Spain.
19. Song, J., Yang, Y., Yang, Y., Huang, Z., and Shen, H.T. (2013, January 22–27). Inter-media hashing for large-scale retrieval from heterogeneous data sources. Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, New York, NY, USA.10.1145/2463676.2465274 ↗
20. Zhou, J., Ding, G., and Guo, Y. (2014, January 6–11). Latent semantic sparse hashing for cross-modal similarity search. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, Gold Coast, Queensland, Australia.10.1145/2600428.2609610 ↗
21. Su, S., Zhong, Z., and Zhang, C. (November, January 27). Deep Joint-Semantics Reconstructing Hashing for Large-Scale Unsupervised Cross-Modal Retrieval. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00312 ↗
22. Yu, J., Zhou, H., Zhan, Y., and Tao, D. (2020, January 7–12). Deep Graph-neighbor Coherence Preserving Network for Unsupervised Cross-modal Hashing. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
23. Liu, S., Qian, S., Guan, Y., Zhan, J., and Ying, L. (2020, January 25–30). Joint-modal Distribution-based Similarity Hashing for Large-scale Unsupervised Deep Cross-modal Retrieval. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, China.10.1145/3397271.3401086 ↗
24. Yang, D., Wu, D., Zhang, W., Zhang, H., Li, B., and Wang, W. (2020, January 8–11). Deep Semantic-Alignment Hashing for Unsupervised Cross-Modal Retrieval. Proceedings of the 2020 International Conference on Multimedia Retrieval, Dublin, Ireland.10.1145/3372278.3390673 ↗
25. Unsupervised Contrastive Cross-Modal Hashing
26. Unsupervised multi-perspective fusing semantic alignment for cross-modal hashing retrieval10.1007/s11042-023-18048-0 ↗
27. Guerrero-Contreras, G., Balderas-Díaz, S., Serrano-Fernández, A., and Muñoz, A. (2024, January 9–12). Enhancing Sentiment Analysis on Social Media: Integrating Text and Metadata for Refined Insights. Proceedings of the 2024 International Conference on Intelligent Environments (IE), Málaga, Spain.10.1109/IE61493.2024.10599899 ↗
28. Balderas-Díaz, S., Guerrero-Contreras, G., Ramírez-Vela, M., Toribio-Camuñas, S., Gay, N.C., and Reguera, A.M. (2024, January 25–27). Inclusive Education and Cultural Heritage Access: The LECTPAT Platform Multimodal Online Dictionary. Proceedings of the 2024 International Symposium on Computers in Education (SIIE), Madrid, Spain.10.1109/SIIE63180.2024.10604516 ↗
29. Csurka, G., Dance, C.R., Fan, L., Willamowski, J., and Bray, C. (2004, January 11–14). Visual categorization with bags of keypoints. Proceedings of the Workshop on Statistical Learning in Computer Vision, ECCV, Prague, Czech Republic.
30. Atwood, J., and Towsley, D. (2016, January 5–10). Diffusion-convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain.
31. Jiang, Z.A. (2024, January 22–24). Multi-Scale Contrastive Learning Networks for Graph Anomaly Detection. Proceedings of the 2024 4th Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS), Harbin, China.
32. CLIP-based fusion-modal reconstructing hashing for large-scale unsupervised cross-modal retrieval10.1007/s13735-023-00268-7 ↗
33. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv.
34. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
35. Contrastive Graph Convolutional Networks With Generative Adjacency Matrix10.1109/TSP.2023.3254888 ↗
36. Sohn, K., Berthelot, D., Li, C.L., Zhang, Z., Carlini, N., Cubuk, E.D., Kurakin, A., Zhang, H., and Raffel, C. (2020). FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. arXiv.
37. Cao, Z., Long, M., Wang, J., and Yu, P.S. (2017, January 22–29). HashNet: Deep Learning to Hash by Continuation. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.598 ↗
38. Wang, F., and Liu, H. (2021, January 20–25). Understanding the Behaviour of Contrastive Loss. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00252 ↗
39. Cui, Y., Jia, M., Lin, T.Y., Song, Y., and Belongie, S.J. (2019, January 15–20). Class-Balanced Loss Based on Effective Number of Samples. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00949 ↗
40. Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv.
41. On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval10.1109/TPAMI.2013.142 ↗
42. Huiskes, M.J., and Lew, M.S. (2008, January 30–31). The MIR flickr retrieval evaluation. Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, Vancouver, BC, Canada.10.1145/1460096.1460104 ↗
43. Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., and Zheng, Y. (2009, January 8–10). NUS-WIDE: A real-world web image database from National University of Singapore. Proceedings of the ACM International Conference on Image and Video Retrieval, Santorini, Greece.10.1145/1646396.1646452 ↗
44. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. arXiv.
45. Wang, J., Liu, W., Kumar, S., and Chang, S.F. (2015). Learning to Hash for Indexing Big Data—A Survey. arXiv.