Adapter-Based Vision Transformer for Cross Domain Few-Shot Classification Using Prototypical Networks
Source abstract
Abstract
Cross-domain few-shot learning (CD-FSL) remains challenging in medical imaging, where labeled data are scarce and source–target domain gaps are often large due to modality differences. In particular, existing few-shot learning methods rely on source–target domain similarity, which limits their effectiveness in cross-modality settings such as MRI-to-CT transfer. To address this problem, this paper proposes an adapter-based Vision Transformer framework for cross-domain few-shot brain tumor classification. Lightweight adapter modules are inserted into a pretrained Vision Transformer to enable parameter-efficient domain adaptation without fine-tuning the entire backbone. In addition, a Prototypical Network is employed to construct class prototypes from limited labeled samples, while a prototype-level Maximum Mean Discrepancy (MMD) loss is introduced to align feature distributions across domains. Unlike prior approaches, the proposed framework introduces a unified prototype-level alignment strategy within an episodic learning paradigm, enabling direct class-wise cross-modal alignment. This design improves generalization under large modality gaps and limited labeled data by jointly optimizing representation learning and domain adaptation. The proposed framework is evaluated on MRI-to-CT brain tumor classification as well as several heterogeneous cross-domain benchmarks, including Chest X-ray, ISIC, CropDisease, and EuroSAT. Experimental results demonstrate that the proposed method achieves competitive performance compared to existing few-shot learning baselines, showing strong robustness under significant domain shifts.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Brain Tumor Detection and Classification · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Dongguk University, Seoul 04620, Republic of Korea
Funding
National Research Foundation of Korea · MSIT · Artificial Intelligence Convergence Innovation Human Resources Development
References (57)
Show source-provided references
1. A deep learning approach for multi-stage classification of brain tumor through magnetic resonance images10.1002/ima.22897 ↗
2. Cross-domain few-shot learning via adaptive transformer networks10.1016/j.knosys.2024.111458 ↗
3. Automated detection of brain tumor through magnetic resonance images using convolutional neural network10.1155/2021/3365043 ↗
4. Guo, Y., Codella, N.C., Karlinsky, L., Codella, J.V., Smith, J.R., Saenko, K., Rosing, T., and Feris, R. (2020, January 23–28). A broader study of cross-domain few-shot learning. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58583-9_8 ↗
5. Phoo, C.P., and Hariharan, B. (2020). Self-training for few-shot transfer across extreme task differences. arXiv.
6. Zhang, T., and Huang, W. (2022, January 23–27). Kernel relative-prototype spectral filtering for few-shot learning. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20044-1_31 ↗
7. Finn, C., Abbeel, P., and Levine, S. (2017, January 6–11). Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
8. Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J.B., and Isola, P. (2020, January 23–28). Rethinking few-shot image classification: A good embedding is all you need?. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58568-6_16 ↗
9. Prototypical networks for few-shot learning
10. Matching networks for one shot learning
11. Lee, K., Maji, S., Ravichandran, A., and Soatto, S. (2019, January 15–20). Meta-learning with differentiable convex optimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01091 ↗
12. Mangla, P., Kumari, N., Sinha, A., Singh, M., Krishnamurthy, B., and Balasubramanian, V.N. (2020, January 1–5). Charting the right manifold: Manifold mixup for few-shot learning. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA.10.1109/WACV45572.2020.9093338 ↗
13. Liu, B., Cao, Y., Lin, Y., Li, Q., Zhang, Z., Long, M., and Hu, H. (2020, January 23–28). Negative margin matters: Understanding margin in few-shot classification. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58548-8_26 ↗
14. Dynamic distillation network for cross-domain few-shot recognition with unlabeled data
15. Yue, X., Zheng, Z., Zhang, S., Gao, Y., Darrell, T., Keutzer, K., and Vincentelli, A.S. (2021, January 19–25). Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.01362 ↗
16. Multi-representation adaptation network for cross-domain image classification10.1016/j.neunet.2019.07.010 ↗
17. Domain-adversarial training of neural networks
18. Long, M., Cao, Y., Wang, J., and Jordan, M. (2015, January 6–11). Learning transferable features with deep adaptation networks. Proceedings of the International Conference on Machine Learning, Lille, France.
19. Saito, K., Kim, D., Sclaroff, S., Darrell, T., and Saenko, K. (2019, January 27–28). Semi-supervised domain adaptation via minimax entropy. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00814 ↗
20. Xu, M., Islam, M., Lim, C.M., and Ren, H. (October, January 27). Class-incremental domain adaptation with smoothing and calibration for surgical report generation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Strasbourg, France.
21. Phuntsho, K., Lee, K., Lee, I., and Ahn, E. (2025). Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions. arXiv.
22. Panareda Busto, P., and Gall, J. (2017, January 22–29). Open set domain adaptation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.88 ↗
23. You, K., Long, M., Cao, Z., Wang, J., and Jordan, M.I. (2019, January 15–20). Universal domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00283 ↗
24. Farahani, A., Voghoei, S., Rasheed, K., and Arabnia, H.R. (2021). A brief review of domain adaptation. Advances in Data Science and Information Engineering, Springer.10.1007/978-3-030-71704-9_65 ↗
25. CDCNet: Cross-domain few-shot learning with adaptive representation enhancement10.1016/j.patcog.2025.111382 ↗
26. Analysis of representations for domain adaptation
27. Gull, S., and Kim, J. (2025). Metric-Based Meta-Learning Approach for Few-Shot Classification of Brain Tumors Using Magnetic Resonance Images. Electronics, 14.10.3390/electronics14091863 ↗
28. Fu, Y., Fu, Y., and Jiang, Y.-G. (2021, January 20–24). Meta-fdmixup: Cross-domain few-shot learning guided by labeled target data. Proceedings of the 29th ACM International Conference on Multimedia, Chengdu, China.10.1145/3474085.3475655 ↗
29. Dziugaite, G.K., Roy, D.M., and Ghahramani, Z. (2015). Training generative neural networks via maximum mean discrepancy optimization. arXiv.
30. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S.Q. (2024). Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv.
31. Enhanced transfer learning method for rolling bearing fault diagnosis based on linear superposition network10.1016/j.engappai.2023.105970 ↗
32. Dynamic joint distribution alignment network for bearing fault diagnosis under variable working conditions10.1109/TIM.2021.3055786 ↗
33. Residual deep subdomain adaptation network: A new method for intelligent fault diagnosis of bearings across multiple domains10.1016/j.mechmachtheory.2021.104635 ↗
34. Multi-branch feature transformation cross-domain few-shot learning for hyperspectral image classification10.1016/j.patcog.2024.111197 ↗
35. Self-supervised learning-based dual-classifier domain adaptation model for rolling bearings cross-domain fault diagnosis10.1016/j.knosys.2023.111229 ↗
36. Li, W.-H., Liu, X., and Bilen, H. (2021, January 10–17). Universal representation learning from multiple domains for few-shot classification. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00939 ↗
37. Mutlu, A., Doğan, Ş., and Tuncer, T. (2025). ViT-ProtoNet for few-shot image classification: A multi-benchmark evaluation. arXiv.
38. Prototypical few-shot learning for histopathology classification: Leveraging foundation models with adapter architectures10.1109/ACCESS.2025.3570673 ↗
39. Cross-data set hyperspectral image classification based on deep domain adaptation10.1109/TGRS.2019.2931730 ↗
40. Jakob, N., and Gurevych, I. (2010, January 9–11). Extracting opinion targets in a single and cross-domain setting with conditional random fields. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Cambridge, MA, USA.
41. Sekhar, A., Bhattacharya, A., Goyal, V., Goel, V., Bhangale, A., Gupta, R.K., and Sethi, A. (2024, January 21–23). Cross-domain evaluation of few-shot classification models: Natural images vs. histopathological images. Proceedings of the International Joint Conference on Biomedical Engineering Systems and Technologies, Rome, Italy.10.1007/978-3-031-96899-0_9 ↗
42. Dual adaptive representation alignment for cross-domain few-shot learning10.1109/TPAMI.2023.3272697 ↗
43. Cai, J., and Shen, S.M. (2020). Cross-domain few-shot learning with meta fine-tuning. arXiv.
44. Enhancing few-shot image classification through learnable multi-scale embedding and attention mechanisms10.1016/j.neunet.2025.107339 ↗
45. Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., and Qiao, Y. (2022). Vision transformer adapter for dense predictions. arXiv.
46. Murtoza, M.G. (2025, September 10). Brain Tumor Multimodal Image (CT & MRI). Available online: https://www.kaggle.com/datasets/murtozalikhon/brain-tumor-multimodal-image-ct-and-mri.
47. Arjunashok33 (2025, October 15). miniImageNet. Available online: https://www.kaggle.com/datasets/arjunashok33/miniimagenet.
48. Ashok, A. (2025, October 22). tieredImageNet. Available online: https://www.kaggle.com/datasets/arjun2000ashok/tieredimagenet.
49. National-Institutes-of-Health-Chest-X-Ray-Dataset (2025, November 05). NIH Chest X-rays. Available online: https://www.kaggle.com/datasets/nih-chest-xrays/data.
50. Chdlr (2025, November 12). ISIC2018 Challenge. Available online: https://www.kaggle.com/datasets/tschandl/isic2018-challenge-task1-data-segmentation.
51. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification10.1109/JSTARS.2019.2918242 ↗
52. Mohanty, S.P., Hughes, D.P., and Salathé, M. (2016). Using deep learning for image-based plant disease detection. Front. Plant Sci., 7.10.3389/fpls.2016.01419 ↗
53. Antoniou, A., Edwards, H., and Storkey, A. (2018). How to train your MAML. arXiv.
54. Hu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y. (2018, January 18–22). Relation networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00378 ↗
55. Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., and Hospedales, T.M. (2018, January 18–22). Learning to compare: Relation network for few-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00131 ↗
56. Guo, Y., Codella, N.C., Karlinsky, L., Smith, J.R., Rosing, T., and Feris, R. (2019). A new benchmark for evaluation of cross-domain few-shot learning. arXiv.10.1007/978-3-030-58583-9_8 ↗
57. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13–18). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning, Virtual.