Zero-Shot Image Classification with Rectified Embedding Vectors Using a Caption Generator
Source abstract
Abstract
Although image recognition technologies are developing rapidly with deep learning, conventional recognition models trained by supervised learning with class labels do not work well when test inputs from untrained classes are given. For example, a recognizer trained to classify Asian bird species cannot recognize the species of kiwi, because the class label “kiwi” and its image samples have not been seen during training. To overcome this limitation, zero-shot classification has been studied recently, and the joint-embedding-based approach has been suggested as one of the promised solutions. In this approach, image features and text descriptions belonging to the same class are trained to be closely located in a common joint-embedding space. Once we obtain the embedding function that can represent the semantic relationship of image–text pairs in training data, test images and text descriptions (prototypes) of unseen classes can also be mapped to the joint-embedding space for classification. The main challenge with this approach is mapping inputs of two different modalities into a common space, and previous works suffer from the inconsistency between the distribution of two feature sets on joint-embedding space extracted from the heterogeneous inputs. To treat this problem, we propose a novel method of employing additional textual information to rectify the visual representation of input images. Since the conceptual information of test classes is generally given as texts, we expect that the additional descriptions from a caption generator can adjust the visual feature for better matching with the representation of the test classes. We also propose to use the generated textual descriptions to augment training samples for learning joint-embedding space. In the experiments on two benchmark datasets, the proposed method shows significant performance improvements of 1.4% on the CUB dataset and 5.5% on the flower dataset, in comparison to existing models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Cancer-related molecular mechanisms research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Kyungpook National University, Daegu 41566, Republic of Korea
Funding
Human Resources Program in Energy Technology of the Korea Institute of Energy Technology Evaluation and Planning(KETEP) · Institute of Information & communications Technology Planning & Evaluation (IITP)
References (43)
Show source-provided references
1. Attribute-based classification for zero-shot visual object categorization10.1109/TPAMI.2013.140 ↗
2. Larochelle, H., Erhan, D., and Bengio, Y. (2008, January 13–17). Zero-data learning of new tasks. Proceedings of the 23rd National Conference on Artificial Intelligence, Chicago, IL, USA.
3. Rohrbach, M., Stark, M., and Schiele, B. (2011, January 20–25). Evaluating knowledge transfer and zero-shot learning in a large-scale setting. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995627 ↗
4. Yu, X., and Aloimonos, Y. (2010, January 5–11). Attribute-Based Transfer Learning for Object Categorization with Zero/One Training Example. Proceedings of the 11th European Conference on Computer Vision, Heraklion, Crete, Greece.10.1007/978-3-642-15555-0_10 ↗
5. Xu, X., Shen, F., Yang, Y., Zhang, D., Shen, H.T., and Song, J. (2017, January 21–26). Matrix Tri-Factorization with Manifold Regularizations for Zero-Shot Learning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.217 ↗
6. Ding, Z., Shao, M., and Fu, Y. (2017, January 21–26). Low-Rank Embedded Ensemble Semantic Dictionary for Zero-Shot Learning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.636 ↗
7. Zhang, L., Xiang, T., and Gong, S. (2017, January 22–25). Learning a deep embedding model for zero-shot learning. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.321 ↗
8. Reed, S., Akata, Z., Lee, H., and Schiele, B. (July, January 26). Learning deep representations of fine-grained visual descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
9. Zhang, Y., and Huchuan, L. (2018, January 8–14). Deep cross-modal projection learning for image-text matching. Proceedings of the 15th European Conference, Munich, Germany (ECCV), Munich, Germany.10.1007/978-3-030-01246-5_42 ↗
10. Li, S., Xiao, T., Li, H., Yang, W., and Wang, X. (2017, January 22–29). Identity-aware textual-visual matching with latent co-attention. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.209 ↗
11. Sarafianos, N., Xu, X., and Kakadiaris, I. (November, January 27). Adversarial Representation Learning for Text-To-Image Matching. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00591 ↗
12. Xu, K., Ba, J.L., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R.S., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML), Lille, France.
13. Understanding and generating simple image descriptions10.1109/TPAMI.2012.162 ↗
14. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
15. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech-Ucsd Birds-200-2011 Dataset, California Institute of Technology.
16. Nilsback, M.-E., and Zisserman, A. (2008, January 16–19). Automated Flower Classification over a Large Number of Classes. Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, Bhubaneswar, India.10.1109/ICVGIP.2008.47 ↗
17. Huang, Y., Wu, Q., Wang, W., and Wang, L. (2018, January 18–23). Learning Semantic Concepts and Order for Image and Sentence Matching. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00645 ↗
18. Deep Relation Embedding for Cross-Modal Retrieval10.1109/TIP.2020.3038354 ↗
19. Lampert, C.H., Nickisch, H., and Harmeling, S. (2009, January 20–25). Learning to Detect Unseen Object Classes by Between-Class Attribute Transfer. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPR.2009.5206594 ↗
20. Farhadi, A., Endres, I., Hoiem, D., and Forsyth, D. (2009, January 20–25). Describing Objects by their Attributes. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPR.2009.5206772 ↗
21. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A ConvNet for the 2020s. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
22. Jayaraman, D., and Grauman, K. (2014, January 8–13). Zero-shot recognition with unreliable attributes. Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), Cambridge, MA, USA.
23. Palatucci, M., Pomerleau, D., Hinton, G., and Mitchell, T.M. (2009, January 7–10). Zero-shot learning with semantic output codes. Proceedings of the 22nd International Conference on Neural Information Processing Systems (NIPS), Vancouver, British.
24. Romera-Paredes, B. (2015, January 6–11). An embarrassingly simple approach to zero-shot learning. Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France.10.1007/978-3-319-50077-5_2 ↗
25. Akata, Z., Reed, S., Walter, D., Lee, H., and Schiele, B. (2015, January 7–12). Evaluation of output embeddings for fine-grained image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298911 ↗
26. Bucher, M., Herbin, S., and Jurie, F. (2016, January 11–14). Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classiffication. Proceedings of the 14th European Conference, Amsterdam, The Netherlands.10.1007/978-3-319-46454-1_44 ↗
27. Alec, R., Jong, W.K., Chris, H., Aditya, R., Gabriel, G., Sandhini, A., Girish, S., Amanda, A., Pamela, M., and Jack, C. (2021). Learning Transferable Visual Models from Natural Language Supervision. arXiv.
28. Li, J., Selvaraju, R.R., Gotmare, A.D., Joty, S., Xiong, C., and Hoi, S. (2021). Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. arXiv.
29. Label-embedding for image classification10.1109/TPAMI.2015.2487986 ↗
30. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. (2013, January 5–10). Devise: A deep visual-semantic embedding model. Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS), Lake Tahoe, NV, USA.
31. Yang, Y., and Hospedales, T. (2014). A unified perspective on multi-domain and multi-task learning. arXiv.
32. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv.
33. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 21–26). Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.345 ↗
34. Chen, H., Ding, G., Lin, Z., Zhao, S., and Han, J. (2018, January 13–19). Show, Observe and Tell: Attribute-Driven Attention Model for Image Captioning. Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden.10.24963/ijcai.2018/84 ↗
35. Cheng, Y., Huang, F., Zhou, L., Jin, C., Zhang, Y., and Zhang, T. (2017, January 7–11). A Hierarchical Multimodal Attention-based Neural Network for Image Captioning. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Tokyo, Japan.10.1145/3077136.3080671 ↗
36. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–23). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
37. Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. (2022). OFA: Unifying Architectures, Tasks, and Modalities through a Simple Sequence-to-Sequence Learning Framework. arXiv.
38. Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., and Cao, Z. (2022). mPLUG: Effective and Efficient Vision-Language Learning by Cross-Modal Skip-Connections. arXiv.10.18653/v1/2022.emnlp-main.488 ↗
39. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., and Gomez, A.N. (2017). Lukasz Kaiser, Illia Polosukhin Attention Is All You Need. arXiv.
40. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
41. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), Montreal, QU, Canada.
42. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
43. Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y. (2020, January 7–12). Random Erasing Data Augmentation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.