VL-Few: Vision Language Alignment for Multimodal Few-Shot Meta Learning
Source abstract
Abstract
Complex tasks in the real world involve different modal models, such as visual question answering (VQA). However, traditional multimodal learning requires a large amount of aligned data, such as image text pairs, and constructing a large amount of training data is a challenge for multimodal learning. Therefore, we propose VL-Few, which is a simple and effective method to solve the multimodal few-shot problem. VL-Few (1) proposes the modal alignment, which aligns visual features into language space through a lightweight model network and improves the multimodal understanding ability of the model; (2) adopts few-shot meta learning in the multimodal problem, which constructs a few-shot meta task pool to improve the generalization ability of the model; (3) proposes semantic alignment to enhance the semantic understanding ability of the model for the task, context, and demonstration; (4) proposes task alignment that constructs training data into the target task form and improves the task understanding ability of the model; (5) proposes generation alignment, which adopts the token-level training and multitask fusion loss to improve the generation ability of the model. Our experimental results show the effectiveness of VL-Few for multimodal few-shot problems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Applied Sciences, Macao Polytechnic University, Macao 999078, China
Funding
Macao Polytechnic University
References (61)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
2. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training, OpenAI.
3. Language models are unsupervised multitask learners
4. Exploring the limits of transfer learning with a unified text-to-text transformer
5. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., and Xia, X. (2022). Glm-130b: An open bilingual pre-trained model. arXiv.
6. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). Opt: Open pre-trained transformer language models. arXiv.
7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
8. Koromilas, P., and Giannakopoulos, T. (2021). Deep multimodal emotion recognition on human speech: A review. Appl. Sci., 11.10.3390/app11177962 ↗
9. Wang, J., Mao, H., and Li, H. (2022). FMFN: Fine-grained multimodal fusion networks for fake news detection. Appl. Sci., 12.10.3390/app12031093 ↗
10. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
11. Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.Y., Wang, Y.X., and Yang, Y. (2023). Aligning large multimodal models with factually augmented rlhf. arXiv.
12. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv.
13. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen technical report. arXiv.
14. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2023). GPT-4 Technical Report. arXiv.
15. Factors affecting innovation resistance of smartphone AI voice assistants10.1080/10447318.2022.2080899 ↗
16. Training language models to follow instructions with human feedback
17. Rani, S., Chauhan, M., Kataria, A., and Khang, A. (2023). Towards the Integration of IoT, Cloud and Big Data: Services, Applications and Standards, Springer.
18. Oladimeji, D., Gupta, K., Kose, N.A., Gundogan, K., Ge, L., and Liang, F. (2023). Smart transportation: An overview of technologies and applications. Sensors, 23.10.3390/s23083880 ↗
19. Kümpel, M., Dech, J., Hawkin, A., and Beetz, M. (June, January 29). Robotic Shopping Assistance for Everyone: Dynamic Query Generation on a Semantic Digital Twin as a Basis for Autonomous Shopping Assistance. Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, London, UK.
20. Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., and Liao, K.D. (2024, January 1–10). A survey on multimodal large language models for autonomous driving. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACVW60836.2024.00106 ↗
21. Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., and Beyer, L. (2022). Pali: A jointly-scaled multilingual language-image model. arXiv.
22. Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O.K., Singhal, S., and Som, S. (2022). Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv.10.1109/CVPR52729.2023.01838 ↗
23. Li, J., Li, D., Savarese, S., and Hoi, S. (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv.
24. Zhang, X., Zeng, Y., Zhang, J., and Li, H. (2023). Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks. arXiv.10.18653/v1/2023.findings-emnlp.40 ↗
25. Chen, S., He, X., Guo, L., Zhu, X., Wang, W., Tang, J., and Liu, J. (2023). Valor: Vision-audio-language omni-perception pretraining model and dataset. arXiv.
26. Wang, P., Wang, S., Lin, J., Bai, S., Zhou, X., Zhou, J., Wang, X., and Zhou, C. (2023). ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities. arXiv.
27. Liu, S., Fan, L., Johns, E., Yu, Z., Xiao, C., and Anandkumar, A. (2023). Prismer: A vision-language model with an ensemble of experts. arXiv.
28. Hu, Y., Stretcu, O., Lu, C.T., Viswanathan, K., Hata, K., Luo, E., Krishna, R., and Fuxman, A. (2023). Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models. arXiv.
29. Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., and Yu, T. (2023). Palm-e: An embodied multimodal language model. arXiv.
30. Yu, Z., Ouyang, X., Shao, Z., Wang, M., and Yu, J. (2023). Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering. arXiv.
31. Lu, J., Gan, R., Zhang, D., Wu, X., Wu, Z., Sun, R., Zhang, J., Zhang, P., and Song, Y. (2023). Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects. arXiv.
32. Park, S., Whang, T., Yoon, Y., and Lim, H. (2021). Multi-view attention network for visual dialog. Appl. Sci., 11.10.3390/app11073009 ↗
33. Meta-learning how to forecast time series10.1002/for.2963 ↗
34. Zhang, Y., Gong, K., Zhang, K., Li, H., Qiao, Y., Ouyang, W., and Yue, X. (2023). Meta-transformer: A unified framework for multimodal learning. arXiv.
35. Shakeel, M., Itoyama, K., Nishida, K., and Nakadai, K. (2023, January 17–20). Metric-based multimodal meta-learning for human movement identification via footstep recognition. Proceedings of the IEEE/SICE International Symposium on System Integration (SII), Atlanta, GA, USA.10.1109/SII55687.2023.10039089 ↗
36. Multimodal fusion methods with deep neural networks and meta-information for aggression detection in surveillance10.1016/j.eswa.2022.118523 ↗
37. Leveraging enhanced task embeddings for generalization in multimodal meta-learning10.1007/s00521-023-08263-z ↗
38. Ma, H., and Yang, K. (2023). MetaSTNet: Multimodal Meta-learning for Cellular Traffic Conformal Prediction. IEEE Trans. Netw. Sci. Eng., 1–14.10.1109/TNSE.2023.3335626 ↗
39. Lin, Z., Yu, S., Kuang, Z., Pathak, D., and Ramanan, D. (2023, January 17–24). Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01852 ↗
40. Moor, M., Huang, Q., Wu, S., Yasunaga, M., Dalmia, Y., Leskovec, J., Zakka, C., Reis, E.P., and Rajpurkar, P. (2023, January 10). Med-flamingo: A multimodal medical few-shot learner. Proceedings of the Machine Learning for Health (ML4H)—PMLR, New Orleans, LA, USA.
41. Yang, X., Feng, S., Wang, D., Zhang, Y., and Poria, S. (November, January 29). Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
42. Wanyan, Y., Yang, X., Chen, C., and Xu, C. (2023, January 17–24). Active Exploration of Multimodal Complementarity for Few-Shot Action Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00628 ↗
43. Khiabani, P.J., and Zubiaga, A. (2023). Few-shot learning for cross-target stance detection by aggregating multimodal embeddings. IEEE Trans. Comput. Soc. Syst., 1–10.10.1109/TCSS.2023.3264114 ↗
44. D’Alessandro, M., Alonso, A., Calabrés, E., and Galar, M. (2023). Multimodal Parameter-Efficient Few-Shot Class Incremental Learning. arXiv.10.1109/ICCVW60793.2023.00364 ↗
45. Khoshboresh-Masouleh, M., and Shah-Hosseini, R. (2023). Multimodal few-shot target detection based on uncertainty analysis in time-series images. Drones, 7.10.3390/drones7020066 ↗
46. Wang, P., Guo, Y., and Wang, Y. (2023). Few-shot multi-modal registration with mono-modal knowledge transfer. Biomed. Signal Process. Control, 85.10.1016/j.bspc.2023.104958 ↗
47. Cai, J., Wu, L., Wu, D., Li, J., and Wu, X. (2023). Multi-Dimensional Information Alignment in Different Modalities for Generalized Zero-Shot and Few-Shot Learning. Information, 14.10.3390/info14030148 ↗
48. Transductive few-shot image recognition with ranking-based multi-modal knowledge transfer
49. Ouali, Y., Bulat, A., Matinez, B., and Tzimiropoulos, G. (2023, January 17–24). Black box few-shot adaptation for vision-language models. Proceedings of the IEEE/CVF International Conference on Computer Vision, Vancouver, BC, Canada.10.1109/ICCV51070.2023.01424 ↗
50. Chadha, A., and Jain, V. (2023). Few-shot Multimodal Multitask Multilingual Learning. arXiv.
51. Jiang, G., Xu, M., Xin, S., Liang, W., Peng, Y., Zhang, C., and Zhu, Y. (2023). MEWL: Few-shot multimodal word learning with referential uncertainty. arXiv.
52. Yang, M., Chen, J., and Velipasalar, S. (2023, January 17–24). Cross-Modality Feature Fusion Network for Few-Shot 3D Point Cloud Classification. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Vancouver, BC, Canada.10.1109/WACV56688.2023.00072 ↗
53. Mokady, R., Hertz, A., and Bermano, A.H. (2021). ClipCap: CLIP Prefix for Image Captioning. arXiv.
54. Najdenkoska, I., Zhen, X., and Worring, M. (2023). Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning. arXiv.
55. Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.S. (2023). Next-gpt: Any-to-any multimodal llm. arXiv.
56. Multimodal few-shot learning with frozen language models
57. Kervadec, C., Antipov, G., Baccouche, M., and Wolf, C. (2021). Estimating semantic structure for the VQA answer space. arXiv.
58. Ma, H., Fan, B., Ng, B.K., and Lam, C.T. (2024). VL-Meta: Vision-Language Models for Multimodal Meta-Learning. Mathematics, 12.10.3390/math12020286 ↗
59. Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., and Dollár, P. (2015). Microsoft COCO: Common Objects in Context. arXiv.10.1007/978-3-319-10602-1_48 ↗
60. Visual genome: Connecting language and vision using crowdsourced dense image annotations10.1007/s11263-016-0981-7 ↗
61. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part V 13.10.1007/978-3-319-10602-1_48 ↗