Multimodal prototype fusion network for paper-cut image classification
Source abstract
Abstract
This paper proposes a Multimodal Prototype Fusion Network (MPFN) to address challenges in paper-cut image classification, including artistic abstraction, imbalanced data, and unseen category adaptation. The framework introduces two variants: AMPFN, which dynamically fuses multimodal prototypes via cross-modal attention and residual learning, and IMPFN, a training-free model for rapid deployment. Leveraging CLIP for feature extraction, AMPFN achieves 90.71% accuracy (16-shot) on seen classes, while IMPFN attains 84.98% accuracy (16-shot) on unseen classes without training. Evaluations on paper-cut datasets and public benchmarks (PACS, ArtDL, CUB-200-2011) demonstrate superiority over existing methods. The approach mitigates data imbalance through n-shot prototypes and reduces computational costs via pre-trained features, proving robust in fine-grained and abstract art classification. This work offers a scalable solution for cultural heritage digitization and multimodal art analysis.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Advanced Neural Network Applications · Image Retrieval and Classification Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (29)
Show source-provided references
1. He K., Zhang X., Ren S.& Sun J. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: 2016:770–778.10.1109/CVPR.2016.90 ↗
2. L, Wang., Y, Zhang. & J. Feng. On the Euclidean distance of images, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 1334-1339, 2005, https://doi.org/10.1109/TPAMI.2005.16510.1109/TPAMI.2005.165 ↗
3. De Maesschalck, R., Jouan-Rimbaud, D. & Massart, D. L. The Mahalanobis distance. Chemometrics Intell. Lab. Syst. 50, 1–18 (2000).10.1016/S0169-7439(99)00047-7 ↗
4. Salton, G., Wong, A. & Yang, C. S. A vector space model for automatic indexing. Commun. ACM 18, 613–620 (1975).10.1145/361219.361220 ↗
5. Vinyals O., Blundell C., Lillicrap T., Kavukcuoglu K.& Wierstra D. Matching networks for one shot learning. arXiv:1606.04080; 2016. https://doi.org/10.48550/arXiv.1606.04080.10.48550/arXiv.1606.04080 ↗
6. J. Snell, K.& Swersky, R. Zeml Prototypical networks for few-shot learning. Proceedings of the Conference on Neural Information Processing Systems (NIPS). Long Beach, USA, 2017: 4077-4087.
7. Li, J. & Allinson, N. M. Relevance feedback in content-based image retrieval: A review. Lect. Notes Comput. Sci 7736, 185–196 (2013).
8. A. Andoni, K. D. Ba. & P. Indyk and D. Woodruff. Efficient sketches for earth-mover distance, with applications, 2009 50th Annual IEEE Symposium on Foundations of Computer Science, Atlanta, GA, USA, 2009,324-330, https://doi.org/10.1109/FOCS.2009.25.10.1109/FOCS.2009.25 ↗
9. Earth, A., Lochl, M. & Axhusen, K. W. Graph-theoretical analysis of the Swiss road and railway networks over time. Netw. Spat. Econ. 7, 1–19 (2007).10.1007/s11067-006-9008-1 ↗
10. Xiang, S., Nie, F. & Zhang, C. Learning a Mahalanobis distance metric for data clustering and classification. Pattern Recognit. 41, 3600–3612 (2008).10.1016/j.patcog.2008.05.018 ↗
11. Li, X. et al. BSNet: Bi - Similarity network for Few - shot Fine - grained Image Classification. IEEE Trans. Image Process 30, 1318–1331 (2021).10.1109/TIP.2020.3043128 ↗
12. Fort S. Gaussian Prototypical Networks for Few-Shot Learning on Omniglot. ArXiv abs/1708.02735, 2017(2017-08-09).
13. Ji, Z., Chai, X., Yu, Y., Pang, Y. & Zhang, Z. Improved prototypical networks for few-Shot learning. Pattern Recognit. Lett. 140, 81–87 (2020).10.1016/j.patrec.2020.07.015 ↗
14. Sung F. et al. Learning to compare: Relation network for few-shot learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, UT, USA: 2018:1199–1208.10.1109/CVPR.2018.00131 ↗
15. B. Mocanu, R. Tapu & T. Zaharia. 2023. Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning. Image Vision Comput. 133, C (May 2023). https://doi.org/10.1016/j.imavis.2023.104676.10.1016/j.imavis.2023.104676 ↗
16. Sun Y. et al. Dynamic metric learning: Towards a scalable metric space to accommodate multiple semantic scales. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Piscataway: IEEE, 2021: 5393-5402.10.1109/CVPR46437.2021.00535 ↗
17. Sun Y. et al. Circle Loss: A unified perspective of pair similarity optimization, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, 6397-6406, https://doi.org/10.1109/CVPR42600.2020.00643.10.1109/CVPR42600.2020.00643 ↗
18. Jia C. et al. Scaling up visual and vision-language representation learning with noisy text supervision [EB/OL]. ArXiv abs/2102.05918, 2021.
19. Vaswani A. et al. Attention Is All You Need. arXiv:1706.03762 (2017).
20. Dosovitskiy A. et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv abs/2010.11929, 2020.
21. Zhou K., Yang J., Loy C. C., Liu Z. Learning to prompt for vision-language models. arXiv:2109.01134, (2021).
22. Radford A. et al. Learning transferable visual models from natural language supervision. arXiv:2103.00020, 2021.
23. Zhou K., Yang J., Loy C. C.& Liu Z. Conditional prompt learning for vision - language models. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA: 2022:16795–16804.10.1109/CVPR52688.2022.01631 ↗
24. Khattak M. U., Rasheed H., Maaz M., Khan S.& Khan F. S. MaPLe: Multi-modal prompt learning. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023, 19113-19122.10.1109/CVPR52729.2023.01832 ↗
25. Huang T., Chu J.& Wei F. Unsupervised prompt learning for vision-language models. ArXiv abs/2204.03649, 2022.
26. Khattak M. U. et al. Self-regulating prompts: Foundational model adaptation without forgetting. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023.10.1109/ICCV51070.2023.01394 ↗
27. Khattak M. U., Naeem M. F., Naseer M., Gool L. V.& Tombari faderico. Learning to prompt with text only supervision for vision-language models. ArXiv abs/2401.02418, (2024).
28. Li D., Yang Y., Song Y.& Hospedales T. M. Deeper, broader and artier domain generalization. Proc IEEE Int Conf Comput Vis (ICCV). 2017;5543–5551.10.1109/ICCV.2017.591 ↗
29. Milani, F. & Fraternali, P. A dataset and a convolutional model for iconography classification in paintings. J. Comput. Cult. Herit. (JOCCH) 14, 1–18 (2021).10.1145/3458885 ↗