Image–Text (IT)-Prompt: Prompt-Based Learning Framework Empowered by the Cluster-Based Nearest Class Mean (C-NCM) for Rehearsal-Free Contrastive Language–Image Pretraining (CLIP)-Based Continual Learning
Source abstract
Abstract
The Contrastive Language–Image Pretraining (CLIP) model has demonstrated remarkable zero-shot capabilities through contrastive learning on large-scale text-image datasets, sparking interest in developing continuous learning methods to extend its knowledge while preserving zero-shot performance. However, traditional approaches often modify the pretrained parameters of CLIP, compromising its zero-shot capabilities, and face challenges due to the substantial parameter size of CLIP and lengthy training times. To address these issues, we propose the Image–Text (IT-)Prompt method, which leverages the inherent correlation between visual and textual information to train discrete prompts dedicated to individual tasks, serving as repositories for task-specific knowledge. By employing discrete textual prompts as guidance, we ensure the uniqueness of each task’s prompt and prevent interference among tasks, thus alleviating catastrophic forgetting during continuous learning. While retaining the pretrained parameters of CLIP, our approach introduces only a small number of additional trainable parameters, enhancing training efficiency and preserving the original zero-shot capabilities of CLIP. Building on IT-Prompt, we further introduce a Cluster-based Nearest Class Mean classifier, which eliminates the need for Softmax classifiers to store and retrain old task samples, significantly improving training efficiency and reducing resource consumption. Experiments demonstrate that our method achieves over a 10% performance improvement compared to state-of-the-art CLIP-based continuous learning methods, with enhanced efficiency and reduced overhead.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Machine Learning and ELM
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Communication Engineering, Communication University of China, Beijing 100024, China
State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
Funding
National Social Science Fund of China
References (53)
Show source-provided references
1. Zhou, D.W., Wang, Q.W., Qi, Z.H., Ye, H.J., Zhan, D.C., and Liu, Z. (2023). Deep class-incremental learning: A survey. arXiv.
2. Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. (2022, January 18–24). Learning to prompt for continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00024 ↗
3. Garg, S., Farajtabar, M., Pouransari, H., Vemulapalli, R., Mehta, S., Tuzel, O., Shankar, V., and Faghri, F. (2023). Tic-clip: Continual training of clip models. arXiv.
4. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
5. Learning to prompt for vision-language models10.1007/s11263-022-01653-1 ↗
6. Ke, Z., and Liu, B. (2022). Continual learning of natural language processing tasks: A survey. arXiv.
7. Lee, K.Y., Zhong, Y., and Wang, Y.X. (2023, January 2–7). Do pre-trained models benefit equally in continual learning?. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV56688.2023.00642 ↗
8. Replay in deep learning: Current approaches and missing biological elements
9. Mai, Z., Li, R., Kim, H., and Sanner, S. (2021, January 20–25). Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00398 ↗
10. Pellegrini, L., Graffieti, G., Lomonaco, V., and Maltoni, D. (2020, January 25–29). Latent replay for real-time continual learning. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341460 ↗
11. Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. (2019). Experience replay for continual learning. Advances in Neural Information Processing Systems, Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019, Curran Associates Inc.
12. Hayes, T.L., and Kanan, C. (2021, January 20–25). Selective replay enhances learning in online continual analogical reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00389 ↗
13. Prototype-guided memory replay for continual learning10.1109/TNNLS.2023.3246049 ↗
14. Ding, Y., Liu, L., Tian, C., Yang, J., and Ding, H. (2022). Don’t stop learning: Towards continual learning for the clip model. arXiv.
15. Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X., and You, Y. (2023, January 4–6). Preventing zero-shot transfer degradation in continual learning of vision-language models. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01752 ↗
16. Tang, Y.M., Peng, Y.X., and Zheng, W.S. (2023, January 4–6). When prompt-based incremental learning does not meet strong pretraining. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00164 ↗
17. Wang, R., Duan, X., Kang, G., Liu, J., Lin, S., Xu, S., Lü, J., and Zhang, B. (2023, January 17–24). Attriclip: A non-incremental learner for incremental knowledge learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00356 ↗
18. Wang, L., Xie, J., Zhang, X., Huang, M., Su, H., and Zhu, J. (2024). Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 10–16 December 2024, Curran Associates Inc.
19. Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.Y., Ren, X., Su, G., Perot, V., and Dy, J. (2022, January 23–27). Dualprompt: Complementary prompting for rehearsal-free continual learning. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19809-0_36 ↗
20. Khan, M.G.Z.A., Naeem, M.F., Van Gool, L., Stricker, D., Tombari, F., and Afzal, M.Z. (2023, January 4–6). Introducing language guidance in prompt-based continual learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01053 ↗
21. Smith, J.S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. (2023, January 4–6). Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Paris, France.10.1109/CVPR52729.2023.01146 ↗
22. Overcoming catastrophic forgetting in neural networks10.1073/pnas.1611835114 ↗
23. Learning without forgetting10.1109/TPAMI.2017.2773081 ↗
24. Shi, Y., Zhou, K., Liang, J., Jiang, Z., Feng, J., Torr, P.H., Bai, S., and Tan, V.Y. (2022, January 18–24). Mimicking the oracle: An initial phase decorrelation approach for class incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01622 ↗
25. Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. (2018, January 8–14). Memory aware synapses: Learning what (not) to forget. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_9 ↗
26. Continual learning of context-dependent processing in neural networks10.1038/s42256-019-0080-x ↗
27. MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning10.1109/TPAMI.2021.3133897 ↗
28. Arani, E., Sarfraz, F., and Zonooz, B. (2022). Learning fast, learning slow: A general continual learning method based on complementary learning system. arXiv.
29. Van De Ven, G.M., Li, Z., and Tolias, A.S. (2021, January 20–25). Class-incremental learning with generative classifiers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00400 ↗
30. Zhou, D.W., Wang, F.Y., Ye, H.J., Ma, L., Pu, S., and Zhan, D.C. (2022, January 18–24). Forward compatible few-shot class-incremental learning. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00884 ↗
31. Wang, F.Y., Zhou, D.W., Liu, L., Ye, H.J., Bian, Y., Zhan, D.C., and Zhao, P. (2022, January 25–29). Beef: Bi-compatible class-incremental learning via energy-based expansion and fusion. Proceedings of the Eleventh International Conference on Learning Representations, Virtual.
32. Li, X., Zhou, Y., Wu, T., Socher, R., and Xiong, C. (2019, January 9–15). Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
33. Konishi, T., Kurokawa, M., Ono, C., Ke, Z., Kim, G., and Liu, B. (2023, January 23–29). Parameter-level soft-masking for continual learning. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
34. Brain-inspired replay for continual learning with artificial neural networks10.1038/s41467-020-17866-2 ↗
35. Van de Ven, G.M., and Tolias, A.S. (2018). Generative replay with feedback connections as a general strategy for continual learning. arXiv.
36. Bagus, B., and Gepperth, A. (2021, January 18–22). An investigation of replay-based approaches for continual learning. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9533862 ↗
37. Tiwari, R., Killamsetty, K., Iyer, R., and Shenoy, P. (2022, January 18–24). Gcr: Gradient coreset based replay buffer selection for continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00020 ↗
38. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning
39. Thengane, V., Khan, S., Hayat, M., and Khan, F. (2022). Clip model is an efficient continual learner. arXiv.
40. Clap4clip: Continual learning with probabilistic finetuning for vision-language models
41. Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., and He, Y. (2024, January 16–22). Boosting continual learning of vision-language models via mixture-of-experts adapters. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02191 ↗
42. Pan, Y., Yuan, Z., Wu, X., Li, Z., and Xu, C. (2024, January 3–6). TMM-CLIP: Task-guided Multi-Modal Alignment for Rehearsal-Free Class Incremental Learning. Proceedings of the 6th ACM International Conference on Multimedia in Asia, Auckland, New Zealand.10.1145/3696409.3700182 ↗
43. Hou, S., Pan, X., Loy, C.C., Wang, Z., and Lin, D. (2019, January 15–20). Learning a unified classifier incrementally via rebalancing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00092 ↗
44. Van de Ven, G.M., and Tolias, A.S. (2019). Three scenarios for continual learning. arXiv.
45. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
46. Li, X.L., and Liang, P. (2021, January 1–6). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.10.18653/v1/2021.acl-long.353 ↗
47. Lin, H., Zhang, B., Feng, S., Li, X., and Ye, Y. (2023, January 17–24). PCR: Proxy-based contrastive replay for online class-incremental continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02322 ↗
48. Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., and Guo, M. (2021, January 11–17). The many faces of robustness: A critical analysis of out-of-distribution generalization. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00823 ↗
49. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech-UCSD Birds-200-2011 Dataset, California Institute of Technology.
50. Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto.
51. Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. (2021). ibot: Image bert pre-training with online tokenizer. arXiv.
52. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021, January 11–17). Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00951 ↗
53. Chen, X., Xie, S., and He, K. (2021, January 11–17). An empirical study of training self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00950 ↗