Supervised Contrastive Learning for 3D Cross-Modal Retrieval
Source abstract
Abstract
Interoperability between different virtual platforms requires the ability to search and transfer digital assets across platforms. Digital assets in virtual platforms are represented in different forms or modalities, such as images, meshes, and point clouds. The cross-modal retrieval of three-dimensional (3D) object representations is challenging due to data representation diversity, making common feature space discovery difficult. Recent studies have been focused on obtaining feature consistency within the same classes and modalities using cross-modal center loss. However, center features are sensitive to hyperparameter variations, making cross-modal center loss susceptible to performance degradation. This paper proposes a new 3D cross-modal retrieval method that uses cross-modal supervised contrastive learning (CSupCon) and the fixed projection head (FPH) strategy. Contrastive learning mitigates the influence of hyperparameters by maximizing feature distinctiveness. The FPH strategy prevents gradient updates in the projection network, enabling the focused training of the backbone networks. The proposed method shows a mean average precision (mAP) increase of 1.17 and 0.14 in 3D cross-modal object retrieval experiments using ModelNet10 and ModelNet40 datasets compared to state-of-the-art (SOTA) methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · 3D Shape Modeling and Analysis · Robotics and Sensor-Based Localization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Contents Convergence Research Center, Korea Electronics Technology Institute (KETI), Seoul 03924, Republic of Korea
Artificial Intelligence Research Center, Korea Electronics Technology Institute (KETI), Seongnam 13509, Republic of Korea
Funding
Ministry of Culture, Sports and Tourism
References (24)
Show source-provided references
1. Yang, L., Ni, S.-T., Wang, Y., Yu, A., Lee, J.-A., and Hui, P. (2024). Interoperability of the metaverse: A digital ecosystem perspective review. arXiv.10.2139/ssrn.4929167 ↗
2. Feng, Y., Zhu, H., Peng, D., Peng, X., and Hu, P. (2023, January 18–22). RONO: Robust Discriminative Learning with Noisy Labels for 2D-3D Cross-Modal Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01117 ↗
3. Jing, L., Vahdani, E., Tan, J., and Tian, Y. (2021, January 19–25). Cross-Modal Center Loss for 3D Cross-Modal Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00316 ↗
4. Wen, Y., Zhang, K., Li, Z., and Qiao, Y. (2016, January 11–14). A Discriminative Feature Learning Approach for Deep Face Recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46478-7_31 ↗
5. Wang, B., Yang, Y., Xu, X., Hanjalic, A., and Shen, H.T. (2017, January 23–27). Adversarial Cross-Modal Retrieval. Proceedings of the 25th ACM International Conference on Multimedia, New York, NY, USA.10.1145/3123266.3123326 ↗
6. HCMSL: Hybrid Cross-Modal Similarity Learning for Cross-Modal Retrieval
7. Zhen, L., Hu, P., Wang, X., and Peng, D. (2019, January 17–20). Deep supervised cross-modal retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01064 ↗
8. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 12–18). A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria.
9. Chen, X., and He, K. (2021, January 19–25). Exploring Simple Siamese Representation Learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01549 ↗
10. Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z., and Azar, M.G. (2020, January 6–12). Bootstrap Your Own Latent—A New Approach to Self-Supervised Learning. Proceedings of the Advances in Neural Information Processing Systems, Online.
11. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 14–19). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online.10.1109/CVPR42600.2020.00975 ↗
12. Afham, M., Dissanayake, I., Dissanayake, D., Dharmasiri, A., Thilakarathna, K., and Rodrigo, R. (2022, January 19–24). CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00967 ↗
13. Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., and Faieta, B. (2021, January 19–25). Multimodal Contrastive Training for Visual Representation Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00692 ↗
14. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. (2020, January 6–12). Supervised Contrastive Learning. Proceedings of the Advances in Neural Information Processing Systems, Online.
15. Cross-domain structure preserving projection for heterogeneous domain adaptation10.1016/j.patcog.2021.108362 ↗
16. Multiview latent space learning with progressively fine-tuned deep features for unsupervised domain adaptation10.1016/j.ins.2024.120223 ↗
17. Semantic Pre-Alignment and Ranking Learning with Unified Framework for Cross-Modal Retrieval10.1109/TCSVT.2022.3182549 ↗
18. Cross-Modal Retrieval with CNN Visual Features: A New Baseline
19. An Orthogonal Subspace Decomposition Method for Cross-Modal Retrieval10.1109/MIS.2022.3169884 ↗
20. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. (2015, January 7–12). 3D Shapenets: A Deep Representation for Volumetric Shapes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
21. Feng, Y., Feng, Y., You, H., Zhao, X., and Gao, Y. (February, January 27). Meshnet: Mesh neural network for 3D shape representation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
22. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
23. Dynamic Graph CNN for Learning on Point Clouds
24. Visualizing Data using t-SNE