Feature Fusion-Based Cross-Modal Proxy Hashing Retrieval
Source abstract
Abstract
Due to its cost-effective and high-efficiency retrieval advantages, deep hashing has attracted extensive attention in the field of cross-modal retrieval. However, despite significant progress in existing deep cross-modal hashing methods, several limitations persist: they struggle to establish consistent mapping relationships across different modalities, fail to effectively bridge the semantic gap between heterogeneous data, and consequently suffer from semantic information loss and incomplete semantic understanding during cross-modal learning. To address these challenges, this paper proposes a Feature Fusion-based Cross-modal Proxy Hashing (FFCPH) retrieval method. This approach integrates multi-modal semantic information through a feature fusion module to generate discriminative and robust fused features. Furthermore, a novel joint loss function, which comprises cross-modal proxy loss, cross-modal irrelevant loss, and cross-modal consistency loss, is designed to preserve inter-sample similarity ranking accuracy and mitigate the semantic gap across modalities. Experimental results on three widely used benchmark datasets demonstrate that the proposed method significantly outperforms state-of-the-art approaches in retrieval performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Multimodal Machine Learning Applications · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Artificial Intelligence, Shanghai University of Electric Power, Shanghai 201300, China
References (35)
Show source-provided references
1. Cao, M., Li, S., Li, J., Nie, L., and Zhang, M. (2022). Image-text retrieval: A survey on recent research and development. arXiv.10.24963/ijcai.2022/759 ↗
2. Multimodal machine learning: A survey and taxonomy10.1109/TPAMI.2018.2798607 ↗
3. Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G.R., Levy, R., and Vasconcelos, N. (2010, January 25–29). A new approach to cross-modal multimedia retrieval. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.10.1145/1873951.1873987 ↗
4. Liu, H., Wang, R., Shan, S., and Chen, X. (2016, January 27–30). Deep supervised hashing for fast image retrieval. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.227 ↗
5. Wang, K., Yin, Q., Wang, W., Wu, S., and Wang, L. (2016). A comprehensive survey on cross-modal retrieval. arXiv.
6. An overview of cross-media retrieval10.1109/TCSVT.2017.2705068 ↗
7. Lin, Z., Ding, G., Hu, M., and Wang, J. (2015, January 7–12). Semantics-preserving hashing for cross-view retrieval. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299011 ↗
8. Zhang, D., and Li, W.-J. (2014, January 27–31). Large-scale supervised multimodal hashing. Proceedings of the AAAI Conference on Artificial Intelligence, Québec City, QC, Canada.10.1609/aaai.v28i1.8995 ↗
9. Learning discriminative binary codes10.1109/TIP.2017.2676345 ↗
10. Shi, Y., You, X., Zheng, F., Wang, S., and Peng, Q. (2019, January 10–16). Equally-guided discriminative hashing. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2019/662 ↗
11. Semantic deep cross-modal hashing10.1016/j.neucom.2020.02.043 ↗
12. Liong, V.E., Lu, J., Tan, Y.P., and Zhou, J. (2017, January 22–29). Cross-modal deep variational hashing. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.439 ↗
13. Tu, J., Liu, X., Lin, Z., Hong, R., and Wang, M. (2022, January 10–14). Differentiable cross-modal hashing via multimodal transformers. Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, Portugal.10.1145/3503161.3548187 ↗
14. Zhu, L., Cai, L., Song, J., Zhu, X., Zhang, C., and Zhang, S. (2022, January 27–30). MSSPQ: Multiple Semantic Structure-Preserving Quantization for Cross-Modal Retrieval. Proceedings of the 2022 International Conference on Multimedia Retrieval, Newark, NJ, USA.10.1145/3512527.3531417 ↗
15. Bai, K., Gao, P., Chen, K., Nie, X., Li, S., and Li, B. (2024, January 15–19). Mutual compromised multi-feature fusion. Proceedings of the 2024 IEEE International Conference on Multimedia and Expo (ICME), Niagara Falls, ON, Canada.
16. Weiss, Y., Torralba, A., and Fergus, R. (2008, January 8–11). Spectral hashing. Proceedings of the Advances in Neural Information Processing Systems 21, Vancouver, BC, Canada.
17. Kumar, S., and Udupa, R. (2011, January 16–22). Learning hash functions for cross-view similarity search. Proceedings of the IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Catalonia‚ Spain.
18. Song, J., Yang, Y., Huang, Z., and Shen, H.T. (2013, January 22–27). Inter-media hashing. Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, New York, NY, USA.
19. Zhou, J., Ding, G., and Guo, Y. (2014, January 6–11). Latent semantic sparse hashing. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, Gold Coast, Australia.
20. Su, S., Zhong, Z., and Zhang, C. (November, January 27). Deep joint-semantics reconstructing hashing. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
21. Hu, H., Xie, L., Hong, R., and Tian, Q. (2020, January 13–19). Unsupervised knowledge distillation. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.
22. Aggregation-based graph convolutional hashing10.1109/TMM.2021.3053766 ↗
23. Deep graph-neighbor coherence preserving network
24. Correlation-identity reconstruction hashing10.1109/TKDE.2022.3218656 ↗
25. Modality-invariant asymmetric networks
26. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models. Proceedings of the International Conference on Machine Learning, Virtual.
27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA.
28. Xu, C., Chai, Z., Xu, Z., Yuan, C., Fan, Y., and Wang, J. (2022, January 10–14). Hyp2 loss. Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, Portugal.
29. Jiang, Q.Y., and Li, W.J. (2017, January 21–26). Deep cross-modal hashing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.348 ↗
30. Li, C., Deng, C., Li, N., Liu, W., Gao, X., and Tao, D. (2018, January 18–23). Self-supervised adversarial hashing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.
31. Gu, W., Gu, X., Gu, J., Li, B., Xiong, Z., and Wang, W. (2019, January 10–13). Adversary guided asymmetric hashing. Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada.
32. Bai, C., Zeng, C., Ma, Q., Zhang, J., and Chen, S. (2020, January 8–11). Deep adversarial discrete hashing. Proceedings of the 2020 International Conference on Multimedia Retrieval, Dublin, Ireland.
33. Tu, R.C., Mao, X.L., Ji, W., Wei, W., and Huang, H. (2023, January 23–27). Data-aware proxy hashing. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, Taipei, Taiwan.
34. Deep neighborhood-aware proxy hashing10.1145/3643639 ↗
35. Learning to hash for indexing big data—A survey10.1109/JPROC.2015.2487976 ↗