TDCMR: Triplet-Based Deep Cross-Modal Retrieval for Geo-Multimedia Data
Source abstract
Abstract
Mass multimedia data with geographical information (geo-multimedia) are collected and stored on the Internet due to the wide application of location-based services (LBS). How to find the high-level semantic relationship between geo-multimedia data and construct efficient index is crucial for large-scale geo-multimedia retrieval. To combat this challenge, the paper proposes a deep cross-modal hashing framework for geo-multimedia retrieval, termed as Triplet-based Deep Cross-Modal Retrieval (TDCMR), which utilizes deep neural network and an enhanced triplet constraint to capture high-level semantics. Besides, a novel hybrid index, called TH-Quadtree, is developed by combining cross-modal binary hash codes and quadtree to support high-performance search. Extensive experiments are conducted on three common used benchmarks, and the results show the superior performance of the proposed method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques · Data Management and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Guangxi Normal University, Guiling 541004, China
School of Computer Science and Engineering, Central South University, Changsha 410083, China
School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing 210094, China
Department of Computer Science, University of Warwick, Warwick CV4 8UW, UK
School of Information Technology and Electrical Engineering, The University of Queensland, St Lucia, QLD 4072, Australia
References (55)
Show source-provided references
1. Robust hashing for image authentication using SIFT feature and quaternion Zernike moments10.1007/s11042-015-3225-x ↗
2. Efficient region of visual interests search for geo-multimedia data10.1007/s11042-018-6750-6 ↗
3. A novel image encryption algorithm based on bit-plane matrix rotation and hyper chaotic systems10.1007/s11042-019-08273-x ↗
4. Deep hashing neural networks for hyperspectral image feature extraction10.1109/LGRS.2019.2899823 ↗
5. Video-based recipe retrieval10.1016/j.ins.2019.11.033 ↗
6. SLTFNet: A spatial and language-temporal tensor fusion network for video moment retrieval10.1016/j.ipm.2019.102104 ↗
7. Cao, D., Yu, Z., Zhang, H., Fang, J., Nie, L., and Tian, Q. (2019, January 21–25). Video-Based Cross-Modal Recipe Retrieval. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3351067 ↗
8. Zhu, L., Song, J., Yang, Z., Huang, W., Zhang, C., and Yu, W. (2021). DAP2CMH: Deep Adversarial Privacy-Preserving Cross-Modal Hashing. Neural Process. Lett., 1–21.10.1007/s11063-021-10447-4 ↗
9. Zhang, H.L., and Huang, S. (2008, January 27–30). A Novel Image Authentication Robust to Geometric Transformations. Proceedings of the 2008 Congress on Image and Signal Processing, Sanya, China.10.1109/CISP.2008.422 ↗
10. Cross-modal recipe retrieval via parallel-and cross-attention networks learning10.1016/j.knosys.2019.105428 ↗
11. Few-shot deep adversarial learning for video-based person re-identification10.1109/TIP.2019.2940684 ↗
12. Cryptanalysis and enhancement of an image encryption scheme based on a 1-D coupled Sine map10.1007/s11071-020-05654-y ↗
13. Deng, G., Xu, C., Tu, X., Li, T., and Gao, N. (2018, January 26–28). Rapid image retrieval with binary hash codes based on deep learning. Proceedings of the Third International Workshop on Pattern Recognition, Jinan, China.
14. Zhu, L., Zhang, C., Song, J., Liu, L., Zhang, S., and Li, Y. (2021, January 5–9). Multi-Graph Based Hierarchical Semantic Fusion for Cross-Modal Representation. Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China.10.1109/ICME51207.2021.9428194 ↗
15. Zheng, W., Zhu, X., Zhu, Y., and Zhang, S. (2018, January 13–19). Robust Feature Selection on Incomplete Data. Proceedings of the IJCAI, Stockholm, Sweden.10.24963/ijcai.2018/443 ↗
16. HCMSL: Hybrid Cross-modal Similarity Learning for Cross-modal Retrieval
17. Jiang, B., Huang, X., Yang, C., and Yuan, J. (2019, January 10–13). Cross-modal video moment retrieval with spatial and language-temporal attention. Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada.10.1145/3323873.3325019 ↗
18. Robust SVM with adaptive graph learning10.1007/s11280-019-00766-x ↗
19. Adversarial Learning-Based Semantic Correlation Representation for Cross-Modal Retrieval10.1109/MMUL.2020.3015764 ↗
20. Zhu, L., Song, J., Wei, X., Yu, H., and Long, J. (2020). CAESAR: Concept augmentation based semantic representation for cross-modal retrieval. Multimed. Tools Appl., 1–31.10.1007/s11042-020-09983-3 ↗
21. Spectral rotation for deep one-step clustering10.1016/j.patcog.2019.107175 ↗
22. Deep Coattention-Based Comparator for Relative Representation Learning in Person Re-Identification10.1109/TNNLS.2020.2979190 ↗
23. Adam, M., Tomáek, P., Lehejek, J., Trojan, J., and Jnek, T. (2021). The Role of Citizen Science and Deep Learning in Camera Trapping. Sustainability, 13.10.3390/su131810287 ↗
24. Franzen, M., Kloetzer, L., Ponti, M., Trojan, J., and Vicens, J. (2021). Machine Learning in Citizen Science: Promises and Implications. The Science of Citizen Science, Springer.10.1007/978-3-030-58278-4_10 ↗
25. Robust Image Hashing Using Radon Transform and Invariant Features10.13164/re.2016.0556 ↗
26. Robust hashing based on quaternion zernike moments for image authentication10.1145/2978572 ↗
27. Zhang, H., Xiong, C., and Geng, G. (2009, January 22–24). Content based image hashing robust to geometric transformations. Proceedings of the 2009 Second International Symposium on Electronic Commerce and Security, Nanchang, China.10.1109/ISECS.2009.118 ↗
28. Wang, Y. (2020). Survey on deep multi-modal data analytics: Collaboration, rivalry and fusion. arXiv.
29. Inverted linear quadtree: Efficient top k spatial keyword search10.1109/TKDE.2016.2530060 ↗
30. Efficient retrieval of the top-k most relevant spatial web objects10.14778/1687627.1687666 ↗
31. Zhang, D., Chan, C.Y., and Tan, K.L. (2014, January 6–11). Processing spatial keyword query as a top-k aggregation query. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, Gold Coast, Australia.10.1145/2600428.2609562 ↗
32. Efficient processing of top-k spatial preference queries10.14778/1921071.1921076 ↗
33. Zhang, D., Tan, K.L., and Tung, A.K. (2013, January 18–22). Scalable top-k spatial keyword search. Proceedings of the 16th International Conference on Extending Database Technology, Genoa, Italy.10.1145/2452376.2452419 ↗
34. Reverse Spatial Visual Top-k Query10.1109/ACCESS.2020.2968982 ↗
35. Efficient continuous top-k geo-image search on road network10.1007/s11042-018-6633-x ↗
36. TDHPPIR: An Efficient Deep Hashing Based Privacy-Preserving Image Retrieval Method10.1016/j.neucom.2019.11.119 ↗
37. Liong, V.E., Lu, J., Tan, Y., and Zhou, J. (2017, January 22–29). Cross-Modal Deep Variational Hashing. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy.10.1109/ICCV.2017.439 ↗
38. Triplet-Based Deep Hashing Network for Cross-Modal Retrieval10.1109/TIP.2018.2821921 ↗
39. Where-and-when to look: Deep siamese attention networks for video-based person re-identification10.1109/TMM.2018.2877886 ↗
40. Yang, E., Deng, C., Liu, W., Liu, X., Tao, D., and Gao, X. (2017, January 4–9). Pairwise Relationship Guided Deep Hashing for Cross-Modal Retrieval. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10719 ↗
41. An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval10.1109/ACCESS.2019.2940055 ↗
42. Multiview spectral clustering via structured low-rank matrix factorization10.1109/TNNLS.2017.2777489 ↗
43. Jiang, Q.Y., and Li, W.J. (2017, January 21–26). Deep cross-modal hashing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.348 ↗
44. Li, C., Deng, C., Li, N., Liu, W., Gao, X., and Tao, D. (2018, January 18–22). Self-supervised adversarial hashing networks for cross-modal retrieval. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake, UT, USA.10.1109/CVPR.2018.00446 ↗
45. SVS-JOIN: Efficient Spatial Visual Similarity Join for Geo-Multimedia10.1109/ACCESS.2019.2948388 ↗
46. Chatfield, K., Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Return of the devil in the details: Delving deep into convolutional nets. arXiv.10.5244/C.28.6 ↗
47. Huiskes, M.J., and Lew, M.S. (2008, January 30–31). The MIR flickr retrieval evaluation. Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, Vancouver, BC, Canada.10.1145/1460096.1460104 ↗
48. Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., and Zheng, Y. (2009, January 8–10). NUS-WIDE: A real-world web image database from National University of Singapore. Proceedings of the ACM International Conference on Image and Video Retrieval, Santorini Island, Greece.10.1145/1646396.1646452 ↗
49. Kumar, S., and Udupa, R. (2011, January 16–22). Learning hash functions for cross-view similarity search. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, Barcelona, Spain.
50. Wang, D., Gao, X., Wang, X., and He, L. (2015, January 25–31). Semantic topic multimodal hashing for cross-media retrieval. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
51. Bronstein, M.M., Bronstein, A.M., Michel, F., and Paragios, N. (2010, January 13–18). Data fusion through cross-modality metric learning using similarity-sensitive hashing. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5539928 ↗
52. Ding, G., Guo, Y., and Zhou, J. (2014, January 24–27). Collective matrix factorization hashing for multimodal data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.267 ↗
53. Zhang, D., and Li, W.J. (2014, January 27–31). Large-scale supervised multimodal hashing with semantic correlation maximization. Proceedings of the AAAI, Québec City, QC, Canada.10.1609/aaai.v28i1.8995 ↗
54. Lin, Z., Ding, G., Hu, M., and Wang, J. (2015, January 7–12). Semantics-preserving hashing for cross-view retrieval. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299011 ↗
55. Zhong, R., Li, G., Tan, K.L., and Zhou, L. (November, January 27). G-tree: An efficient index for KNN search on road networks. Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management, San Francisco, CA, USA.