Proxy-Based Semi-Supervised Cross-Modal Hashing
Source abstract
Abstract
Due to the difficulty in obtaining label information in practical applications, semi-supervised cross-modal retrieval has emerged. However, the existing semi-supervised cross-modal hashing retrieval methods mainly focus on exploring the structural relationships between data and generating high-quality discrete pseudo-labels while neglecting the relationships between data and categories, as well as the structural relationships between data and categories inherent in continuous pseudo-labels. Based on this, Proxy-based Semi-Supervised Cross-Modal Hashing (PSSCH) is proposed. Specifically, we propose a category proxy network to generate category center points in both feature and hash spaces. Additionally, we design an Adaptive Dual-Label Loss function, which applies different learning strategies to discrete ground truth labels and continuous pseudo-labels and adaptively increases the training weights of unlabeled data with more epochs. Experiments on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets show that PSSCH achieves the highest mAP improvements of 3%, 1%, and 4%, respectively, demonstrating better results than the latest baseline methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Algorithms and Data Compression · Advanced Data Compression Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Information and Intelligence, Hunan Agricultural University, Changsha 410128, China
References (55)
Show source-provided references
1. Liu, S., Qian, S., Guan, Y., Zhan, J., and Ying, L. (2020, January 25–30). Joint-modal distribution-based similarity hashing for large-scale unsupervised deep cross-modal retrieval. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event.10.1145/3397271.3401086 ↗
2. Shi, Y., You, X., Zheng, F., Wang, S., and Peng, Q. (2019, January 10–16). Equally-Guided Discriminative Hashing for Cross-modal Retrieval. Proceedings of the IJCAI, Macao, China.10.24963/ijcai.2019/662 ↗
3. Deep Neighborhood-preserving Hashing with Quadratic Spherical Mutual Information for Cross-modal Retrieval10.1109/TMM.2023.3349075 ↗
4. Contrastive Multi-Bit Collaborative Learning for Deep Cross-Modal Hashing10.1109/TKDE.2024.3419577 ↗
5. Deep Ranking Distribution Preserving Hashing for Robust Multi-Label Cross-modal Retrieval10.1109/TMM.2024.3358995 ↗
6. Self-constraining and attention-based hashing network for bit-scalable cross-modal retrieval10.1016/j.neucom.2020.03.019 ↗
7. Liu, Y., Wu, Q., Zhang, Z., Zhang, J., and Lu, G. (November, January 29). Multi-Granularity Interactive Transformer Hashing for Cross-modal Retrieval. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
8. Deep self-enhancement hashing for robust multi-label cross-modal retrieval10.1016/j.patcog.2023.110079 ↗
9. Gao, Z., Wang, J., Yu, G., Yan, Z., Domeniconi, C., and Zhang, J. (2023, January 7–14). Long-tail cross modal hashing. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i6.25927 ↗
10. Oria, V., Sapino, M.L., Satoh, S., Kerhervé, B., Cheng, W., Ide, I., and Singh, V.K. (2022, January 27–30). MSSPQ: Multiple Semantic Structure-Preserving Quantization for Cross-Modal Retrieval. Proceedings of the ICMR ’22: International Conference on Multimedia Retrieval, Newark, NJ, USA.
11. Cross-Domain Transfer Hashing for Efficient Cross-Modal Retrieval10.1109/TCSVT.2024.3374791 ↗
12. Weighted cross-modal hashing with label enhancement10.1016/j.knosys.2024.111657 ↗
13. HCMSL: Hybrid Cross-modal Similarity Learning for Cross-modal Retrieval10.1145/3412847 ↗
14. Fan, W., Zhang, C., Li, H., Jia, X., and Wang, G. (2023). Three-stage semisupervised cross-modal hashing with pairwise relations exploitation. IEEE Trans. Neural Netw. Learn. Syst.
15. Semi-supervised semantic factorization hashing for fast cross-modal retrieval10.1007/s11042-017-4567-3 ↗
16. Semi-supervised discrete hashing for efficient cross-modal retrieval10.1007/s11042-020-09195-9 ↗
17. Weakly supervised cross-modal hashing
18. Yang, L., Zhang, K., Li, Y., Chen, Y., Long, J., and Yang, Z. (2023, January 20–23). S3ACH: Semi-Supervised Semantic Adaptive Cross-Modal Hashing. Proceedings of the International Conference on Neural Information Processing, Changsha, China.10.1007/978-981-99-8070-3_20 ↗
19. Semi-supervised cross-modal hashing with multi-view graph representation10.1016/j.ins.2022.05.006 ↗
20. Shen, X., Yu, G., Chen, Y., Yang, X., and Zheng, Y. (2024–1, January 28). Graph Convolutional Semi-Supervised Cross-Modal Hashing. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, VIC, Australia.10.1145/3664647.3680732 ↗
21. Semi-supervised knowledge distillation for cross-modal hashing10.1109/TMM.2021.3129623 ↗
22. Semi-supervised semi-paired cross-modal hashing10.1109/TCSVT.2023.3312385 ↗
23. Jiang, Q.Y., and Li, W.J. (2017, January 21–26). Deep cross-modal hashing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.348 ↗
24. Multi-label enhancement based self-supervised deep cross-modal hashing10.1016/j.neucom.2021.09.053 ↗
25. HHF: Hashing-guided hinge function for deep hashing retrieval10.1109/TMM.2022.3222598 ↗
26. Specific class center guided deep hashing for cross-modal retrieval10.1016/j.ins.2022.07.095 ↗
27. Deep cross-modal proxy hashing
28. Tu, R.C., Mao, X.L., Ji, W., Wei, W., and Huang, H. (2023, January 23–27). Data-aware proxy hashing for cross-modal retrieval. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, Taipei, Taiwan.10.1145/3539618.3591660 ↗
29. Deep semantic-aware proxy hashing for multi-label cross-modal retrieval10.1109/TCSVT.2023.3285266 ↗
30. Cross-Modal Hashing Method with Properties of Hamming Space: A New Perspective10.1109/TPAMI.2024.3392763 ↗
31. Chen, H., Zhu, L., and Zhu, X. (2024). Deep Class-guided Hashing for Multi-label Cross-modal Retrieval. arXiv.
32. Li, X., Hu, D., and Nie, F. (2017, January 23–27). Deep binary reconstruction for cross-modal hashing. Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, CA, USA.10.1145/3123266.3123355 ↗
33. Wu, G., Lin, Z., Han, J., Liu, L., Ding, G., Zhang, B., and Shen, J. (2018, January 13–19). Unsupervised Deep Hashing via Binary Latent Factor Models for Large-scale Cross-modal Retrieval. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden.10.24963/ijcai.2018/396 ↗
34. Su, S., Zhong, Z., and Zhang, C. (2019, January 27–28). Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00312 ↗
35. Aggregation-based graph convolutional hashing for unsupervised cross-modal retrieval10.1109/TMM.2021.3053766 ↗
36. Work together: Correlation-identity reconstruction hashing for unsupervised cross-modal retrieval10.1109/TKDE.2022.3218656 ↗
37. Semi-supervised cross-modal hashing via modality-specific and cross-modal graph convolutional networks10.1016/j.patcog.2022.109211 ↗
38. A semi-supervised cross-modal memory bank for cross-modal retrieval10.1016/j.neucom.2024.127430 ↗
39. Wang, J., Gong, T., and Yan, Y. (2024, January 14–18). Semi-supervised Prototype Semantic Association Learning for Robust Cross-modal Retrieval. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Washington, DC, USA.10.1145/3626772.3657756 ↗
40. Tu, J., Liu, X., Lin, Z., Hong, R., and Wang, M. (2022, January 10–14). Differentiable cross-modal hashing via multimodal transformers. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3548187 ↗
41. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 4). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
42. Rawat, A., Dua, I., Gupta, S., and Tallamraju, R. (2022, January 23–27). Semi-supervised Domain Adaptation by Similarity Based Pseudo-Label Injection. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-25063-7_10 ↗
43. Abdelfattah, R., Guo, Q., Li, X., Wang, X., and Wang, S. (2023, January 2–6). Cdul: Clip-driven unsupervised learning for multi-label image classification. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00130 ↗
44. Huiskes, M.J., and Lew, M.S. (2008, January 30–31). The mir flickr retrieval evaluation. Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, Vancouver, BC, Canada.10.1145/1460096.1460104 ↗
45. Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., and Zheng, Y. (2009, January 8–10). Nus-wide: A real-world web image database from national university of singapore. Proceedings of the ACM International Conference on Image and Video Retrieval, Santorini Island, Greece.10.1145/1646396.1646452 ↗
46. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part V 13.10.1007/978-3-319-10602-1_48 ↗
47. Language models are unsupervised multitask learners
48. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 13–15). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtually.
49. Wang, Y., Luo, X., and Xu, X.S. (2020, January 12–16). Label embedding online hashing for cross-modal retrieval. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3413971 ↗
50. Enhanced discrete multi-modal hashing: More constraints yet less time to learn10.1109/TKDE.2020.2995195 ↗
51. Hierarchical consensus hashing for cross-modal retrieval10.1109/TMM.2023.3272169 ↗
52. Teacher-student learning: Efficient hierarchical message aggregation hashing for cross-modal retrieval10.1109/TMM.2022.3177901 ↗
53. Yu, J., Zhou, H., Zhan, Y., and Tao, D. (2021, January 2–9). Deep graph-neighbor coherence preserving network for unsupervised cross-modal hashing. Proceedings of the AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v35i5.16592 ↗
54. Unsupervised contrastive cross-modal hashing
55. Visualizing data using t-SNE