Few-Shot Image Classification via Mutual Distillation
Source abstract
Abstract
Due to their compelling performance and appealing simplicity, metric-based meta-learning approaches are gaining increasing attention for addressing the challenges of few-shot image classification. However, many similar methods employ intricate network architectures, which can potentially lead to overfitting when trained with limited samples. To tackle this concern, we propose using mutual distillation to enhance metric-based meta-learning, effectively bolstering model generalization. Specifically, our approach involves two individual metric-based networks, such as prototypical networks and relational networks, mutually supplying each other with a regularization term. This method seamlessly integrates with any metric-based meta-learning approach. We undertake comprehensive experiments on two prevalent few-shot classification benchmarks, namely miniImageNet and Caltech-UCSD Birds-200-2011 (CUB), to demonstrate the effectiveness of our proposed algorithm. The results demonstrate that our method efficiently enhances each metric-based model through mutual distillation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Domain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer and Information, Hohai University, Nanjing 210024, China
School of Electrical Engineering, Nantong University, Nantong 226019, China
Key Laboratory of Intelligent Information Processing, Nanjing Xiaozhuang University, Nanjing 211171, China
Funding
Joint Fund of Ministry of Education for Equipment Pre-research · Research Fund from Science and Technology on Underwater Vehicle Technology Laboratory · Key Laboratory of Information System Requirements · Water Science and Technology Project of Jiangsu Province · Qinglan Project of Jiangsu Province
References (49)
Show source-provided references
1. Imagenet classification with deep convolutional neural networks
2. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
3. Revisiting metric learning for few-shot image classification10.1016/j.neucom.2020.04.040 ↗
4. Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. (2016). One-shot learning with memory-augmented neural networks. arXiv.
5. Munkhdalai, T., and Yu, H. (2017, January 6–11). Meta networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
6. Finn, C., Abbeel, P., and Levine, S. (2017, January 6–11). Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
7. Wang, Z., Zhao, Y., Li, J., and Tian, Y. (2020, January 12–16). Cooperative Bi-path Metric for Few-shot Learning. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3413946 ↗
8. Li, A., Huang, W., Lan, X., Feng, J., Li, Z., and Wang, L. (2022, January 18–24). Boosting few-shot learning with adaptive margin loss. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.
9. Koch, G., Zemel, R., and Salakhutdinov, R. (2015, January 6–11). Siamese neural networks for one-shot image recognition. Proceedings of the ICML Deep Learning Workshop, Lille, France.
10. Matching networks for one shot learning
11. Snell, J., Swersky, K., and Zemel, R.S. (2017). Prototypical networks for few-shot learning. arXiv.
12. Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., and Hospedales, T.M. (2018, January 18–23). Learning to compare: Relation network for few-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00131 ↗
13. Li, H., Eigen, D., Dodge, S., Zeiler, M., and Wang, X. (2019, January 15–20). Finding task-relevant features for few-shot learning by category traversal. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00009 ↗
14. Wang, X., Yu, F., Wang, R., Darrell, T., and Gonzalez, J.E. (2019, January 15–20). Tafe-net: Task-aware feature embeddings for low shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00193 ↗
15. Ye, H.J., Hu, H., Zhan, D.C., and Sha, F. (2019). Learning embedding adaptation for few-shot learning. arXiv.
16. Liu, Y., Lee, J., Park, M., Kim, S., Yang, E., Hwang, S.J., and Yang, Y. (2018). Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv.
17. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
18. Ding, Q., Wu, S., Sun, H., Guo, J., and Xia, S.T. (2019). Adaptive regularization of labels. arXiv.
19. Knowledge distillation: A survey10.1007/s11263-021-01453-z ↗
20. Wu, Z., Li, Y., Guo, L., and Jia, K. (November, January 27). PARN: Position-aware relation networks for few-shot learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
21. Hou, R., Chang, H., Ma, B., Shan, S., and Chen, X. (2019). Cross attention network for few-shot classification. arXiv.
22. Simon, C., Koniusz, P., Nock, R., and Harandi, M. (2020, January 13–19). Adaptive subspaces for few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00419 ↗
23. Wu, F., Smith, J.S., Lu, W., Pang, C., and Zhang, B. (2020, January 23–28). Attentive prototype few-shot learning with capsule network-based embedding. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58604-1_15 ↗
24. Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., and Luo, J. (2019, January 15–19). Revisiting local descriptor based image-to-class measure for few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00743 ↗
25. Bateni, P., Goyal, R., Masrani, V., Wood, F., and Sigal, L. (2020, January 13–19). Improved few-shot visual classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01450 ↗
26. Nguyen, V.N., Løkse, S., Wickstrøm, K., Kampffmeyer, M., Roverso, D., and Jenssen, R. (2020, January 23–28). SEN: A Novel Feature Normalization Dissimilarity Measure for Prototypical Few-Shot Learning Networks. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XXIII 16.10.1007/978-3-030-58592-1_8 ↗
27. Zhang, C., Cai, Y., Lin, G., and Shen, C. (2020, January 13–19). DeepEMD: Few-Shot Image Classification with Differentiable Earth Mover’s Distance and Structured Classifiers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01222 ↗
28. Zhang, Y., Xiang, T., Hospedales, T.M., and Lu, H. (2018, January 18–23). Deep mutual learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00454 ↗
29. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
30. Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. (2017). A simple neural attentive meta-learner. arXiv.
31. Hilliard, N., Phillips, L., Howland, S., Yankov, A., Corley, C.D., and Hodas, N.O. (2018). Few-shot learning with metric-agnostic conditional embeddings. arXiv.
32. Li, Z., Zhou, F., Chen, F., and Li, H. (2017). Meta-sgd: Learning to learn quickly for few-shot learning. arXiv.
33. Nichol, A., Achiam, J., and Schulman, J. (2018). On first-order meta-learning algorithms. arXiv.
34. Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., and Pontil, M. (2018, January 10–15). Bilevel programming for hyperparameter optimization and meta-learning. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
35. Munkhdalai, T., Yuan, X., Mehri, S., and Trischler, A. (2018, January 10–15). Rapid adaptation with conditionally shifted neurons. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
36. Grant, E., Finn, C., Levine, S., Darrell, T., and Griffiths, T. (2018). Recasting gradient-based meta-learning as hierarchical bayes. arXiv.
37. Bayesian meta-learning for the few-shot setting via deep kernels
38. Oh, J., Yoo, H., Kim, C., and Yun, S.Y. (2020). Boil: Towards representation change for few-shot learning. arXiv.
39. Snell, J., and Zemel, R. (2020). Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes. arXiv.
40. Prol, H., Dumoulin, V., and Herranz, L. (2018). Cross-modulation networks for few-shot learning. arXiv.
41. Zhang, H., Zhang, J., and Koniusz, P. (2019, January 15–19). Few-shot learning via saliency-guided hallucination of samples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00288 ↗
42. Hui, B., Zhu, P., Hu, Q., and Wang, Q. (2019, January 8–12). Self-attention relation network for few-shot learning. Proceedings of the 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Shanghai, China.10.1109/ICMEW.2019.00041 ↗
43. Hao, F., He, F., Cheng, J., Wang, L., Cao, J., and Tao, D. (November, January 27). Collect and select: Semantic alignment metric learning for few-shot learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
44. L2AE-D: Learning to aggregate embeddings for few-shot learning with meta-level dropout10.1016/j.neucom.2021.02.024 ↗
45. Li, W., Xu, J., Huo, J., Wang, L., Gao, Y., and Luo, J. (February, January 27). Distribution consistency based covariance metric networks for few-shot learning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
46. Tang, S., Chen, D., Bai, L., Liu, K., Ge, Y., and Ouyang, W. (2021, January 20–25). Mutual CRF-GNN for Few-Shot Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00236 ↗
47. Li, K., Zhang, Y., Li, K., and Fu, Y. (2020, January 13–19). Adversarial feature hallucination networks for few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01348 ↗
48. Ravi, S., and Larochelle, H. (2016, January 2–4). Optimization as a model for few-shot learning. Proceedings of the International Conference on Learning Representations, San Juan, DC, USA.
49. Zhang, H., Torr, P.H., and Koniusz, P. (2020). Few-shot Learning with Multi-scale Self-supervision. arXiv.