Knowledge Distillation Based on Fitting Ground-Truth Distribution of Images
Source abstract
Abstract
Knowledge distillation based on the features from the penultimate layer allows the student (lightweight model) to efficiently mimic the internal feature outputs of the teacher (high-capacity model). However, the training data may not conform to the ground-truth distribution of images in terms of classes and features. We propose two knowledge distillation algorithms to solve the above problem from the directions of fitting the ground-truth distribution of classes and fitting the ground-truth distribution of features, respectively. The former uses teacher labels to supervise student classification output instead of dataset labels, while the latter designs feature temperature parameters to correct teachers’ abnormal feature distribution output. We conducted knowledge distillation experiments on the ImageNet-2012 and Cifar-100 datasets using seven sets of homogeneous models and six sets of heterogeneous models. The experimental results show that our proposed algorithms improve the performance of penultimate layer feature knowledge distillation and outperform other existing knowledge distillation methods in terms of classification performance and generalization ability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Machine Learning and Data Classification · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China
Funding
National Natural Science Foundation of China · Guangxi Natural Science Foundation
References (22)
Show source-provided references
1. Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (2018, January 8–14). Shufflenet v2: Practical guidelines for efficient cnn architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_8 ↗
2. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
3. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
4. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
5. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
6. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
7. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference, Munich, Germany. Proceedings, Part III 18; Lecture Notes in Computer Science.10.1007/978-3-319-24574-4_28 ↗
8. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
9. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
10. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
11. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., and Bengio, Y. (2014). Fitnets: Hints for thin deep nets. arXiv.
12. Tian, Y., Krishnan, D., and Isola, P. (2019). Contrastive representation distillation. arXiv.
13. Distilling knowledge by mimicking features
14. Xu, K., Rui, L., Li, Y., and Gu, L. (2020, January 23–28). Feature normalized knowledge distillation for image classification. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58595-2_40 ↗
15. Classification in the presence of label noise: A survey
16. Zhao, B., Cui, Q., Song, R., Qiu, Y., and Liang, J. (2022, January 18–24). Decoupled knowledge distillation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01165 ↗
17. Zagoruyko, S., and Komodakis, N. (2016). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv.
18. Heo, B., Lee, M., Yun, S., and Choi, J.Y. (February, January 27). Knowledge transfer via distillation of activation boundaries formed by hidden neurons. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
19. Tung, F., and Mori, G. (November, January 27). Similarity-preserving knowledge distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
20. Yim, J., Joo, D., Bae, J., and Kim, J. (2017, January 21–26). A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.754 ↗
21. Kim, J., Park, S.U., and Kwak, N. (2018). Paraphrasing complex network: Network compression via factor transfer. Adv. Neural Inf. Process. Syst., 31.
22. Xu, G., Liu, Z., Li, X., and Loy, C.C. (2020, January 23–28). Knowledge distillation meets self-supervision. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58545-7_34 ↗