TECMH: Transformer-Based Cross-Modal Hashing For Fine-Grained Image-Text Retrieval
Source abstract
Abstract
In recent years, cross-modal hash retrieval has become a popular research field because of its advantages of high efficiency and low storage. Cross-modal retrieval technology can be applied to search engines, cross-modal medical processing, etc. The existing main method is to use a multi-label matching paradigm to finish the retrieval tasks. However, such methods do not use fine-grained information in the multi-modal data, which may lead to sub-optimal results. To avoid cross-modal matching turning into label matching, this paper proposes an end-to-end fine-grained cross-modal hash retrieval method, which can focus more on the fine-grained semantic information of multi-modal data. First, the method refines the image features and no longer uses multiple labels to represent text features but uses BERT for processing. Second, this method uses the inference capabilities of the transformer encoder to generate global fine-grained features. Finally, in order to better judge the effect of the fine-grained model, this paper uses the datasets in the image text matching field instead of the traditional label-matching datasets. This article experiment on Microsoft COCO (MS-COCO) and Flickr30K datasets and compare it with the previous classical methods. The experimental results show that this method can obtain more advanced results in the cross-modal hash retrieval field.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (48)
Show source-provided references
1. F. Faghri, D. J. Fleet, J. R. Kiros and S. Fidler, “VSE++: Improving visual-semantic embeddings with hard negatives,” arXiv preprint, arXiv:170705612, 2017.
2. R. Kiros, R. Salakhutdinov and R. S. Zemel, “Unifying visual-semantic embeddings with multimodal neural language models,” arXiv preprint arXiv:1411.2539, 2014.
3. Devise: A deep visual-semantic embedding model
4. D. Qi, L. Su, J. Song, E. Cui, T. Bharti et al., “ImageBERT: Cross-modal pre-training with large-scale weak-supervised image-text data,” arXiv preprint arXiv:2001.07966, 2020.
5. Stacked cross attention for image-text matching
6. Transformer reasoning network for image-text matching and retrieval
7. ViLBERT: Pre-training task-agnostic visiolinguistic representations for vision-and-language tasks
8. Deep cross-modal hashing
9. Self-supervised adversarial hashing networks for cross-modal retrieval
10. Cross-modal deep variational hashing
11. Large-scale supervised multi-modal hashing with semantic correlation maximization
12. Collective matrix factorization hashing for multimodal data
13. Semantics-preserving hashing for cross-view retrieval
14. Deep supervised cross-modal retrieval
15. Semantic deep cross-modal hashing10.1016/j.neucom.2020.02.043 ↗
16. G. Mikriukov, M. Ravanbakhsh and B. Demir, “Deep unsupervised contrastive hashing for large-scale cross-modal text-image retrieval in remote sensing,” arXiv preprint arXiv:2201.08125, 2022.10.1109/ICASSP43922.2022.9746251 ↗
17. Deep normalized cross-modal hashing with bi-direction relation reasoning
18. Differentiable cross-modal hashing via multimodal transformers
19. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai et al., “An image is worth 16 × 16 words: Transformers for image recognition at scale,” arXiv preprint, arXiv:2010.11929, 2020.
20. Transhash: Transformer-based hamming hashing for efficient image retrieval
21. S. R. Dubey, S. K. Singh and W. T. Chu, “Vision transformer hashing for image retrieval,” arXiv preprint, arXiv:210912564, 2021.10.1109/ICME52920.2022.9859900 ↗
22. Hashformer: Vision transformer based deep hashing for image retrieval10.1109/LSP.2022.3157517 ↗
23. Deep visual-semantic alignments for generating image descriptions
24. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
25. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint, arXiv:14091556, 2014.
26. Automatic concept discovery from parallel text and visual corpora
27. Attention is all you need
28. J. Devlin, M. W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint, arXiv:181004805, 2018.
29. Attention-based LSTM for aspect-level sentiment classification
30. M. T. Luong, H. Pham and C. D. Manning, “Effective approaches to attention-based neural machine translation,” arXiv preprint, arXiv:150804025, 2015.10.18653/v1/D15-1166 ↗
31. CBAM: Convolutional block attention module
32. Faster R-CNN: Towards real-time object detection with region proposal networks
33. Bottom-up and top-down attention for image captioning and visual question answering
34. Visual genome: Connecting language and vision using crowdsourced dense image annotations10.1007/s11263-016-0981-7 ↗
35. Deep metric learning with angular loss
36. Improved deep metric learning with multi-class n-pair loss objective
37. Microsoft COCO: Common objects in context
38. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions10.1162/tacl_a_00166 ↗
39. Efficient fine-grained visual-text search using adversarially-learned hash codes
40. Learning fragment self-attention embeddings for image-text matching
41. SMAN: Stacked multimodal attention network for cross-modal image-text retrieval
42. Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders10.1145/3451390 ↗
43. Cross-modal hamming hashing
44. Self-constraining and attention-based hashing network for bit-scalable cross-modal retrieval10.1016/j.neucom.2020.03.019 ↗
45. Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval
46. Z. Zheng, L. Zheng, M. Garrett, Y. Yang, M. Xu et al., “Dual-path convolutional image-text embeddings with instance loss,” arXiv e-prints, pp. arXiv–1711, 2017.
47. Multi-modality cross attention network for image and sentence matching
48. Learning semantic concepts and order for image and sentence matching