Revisiting Label Smoothing Regularization with Knowledge Distillation
Source abstract
Abstract
Label Smoothing Regularization (LSR) is a widely used tool to generalize classification models by replacing the one-hot ground truth with smoothed labels. Recent research on LSR has increasingly focused on the correlation between the LSR and Knowledge Distillation (KD), which transfers the knowledge from a teacher model to a lightweight student model by penalizing their output’s Kullback–Leibler-divergence. Based on this observation, a Teacher-free Knowledge Distillation (Tf-KD) method was proposed in previous work. Instead of a real teacher model, a handcrafted distribution similar to LSR was used to guide the student learning. Tf-KD is a promising substitute for LSR except for its hard-to-tune and model-dependent hyperparameters. This paper develops a new teacher-free framework LSR-OS-TC, which decomposes the Tf-KD method into two components: model Output Smoothing (OS) and Teacher Correction (TC). Firstly, the LSR-OS extends the LSR method to the KD regime and applies a softer temperature to the model output softmax layer. Output smoothing is critical for stabilizing the KD hyperparameters among different models. Secondly, in the TC part, a larger proportion is assigned to the uniform distribution teacher’s right class to provide a more informative teacher. The two-component method was evaluated exhaustively on the image (dataset CIFAR-100, CIFAR-10, and CINIC-10) and audio (dataset GTZAN) classification tasks. The results showed that LSR-OS can improve LSR performance independently with no extra computational cost, especially on several deep neural networks where LSR is ineffective. The further training boost by the TC component showed the effectiveness of our two-component strategy. Overall, LSR-OS-TC is a practical substitution of LSR that can be tuned on one model and directly applied to other models compared to the original Tf-KD method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image Enhancement Techniques · Advanced Neural Network Applications · Infrastructure Maintenance and Monitoring
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China
School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China
Funding
National Natural Science Foundation of China
References (42)
Show source-provided references
1. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
2. Huang, G., Liu, Z., Pleiss, G., Van Der Maaten, L., and Weinberger, K. (2019). Convolutional Networks with Dense Connectivity. IEEE Trans. Pattern Anal. Mach. Intell.
3. Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., and Feng, J. (2017, January 4–9). Dual path networks. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
4. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
5. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–23). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
6. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for MobileNetV3. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00140 ↗
7. Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (2018, January 8–14). Shufflenet v2: Practical guidelines for efficient cnn architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_8 ↗
8. Liu, H., Simonyan, K., and Yang, Y. (2019). DARTS: Differentiable Architecture Search. arXiv.
9. Tan, M., and Le, Q. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
10. Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H.P. (2016). Pruning Filters for Efficient ConvNets. arXiv.
11. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016, January 11–14). Xnor-net: Imagenet classification using binary convolutional neural networks. Proceedings of the European conference on computer vision, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_32 ↗
12. Han, S., Mao, H., and Dally, W.J. (2015). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv.
13. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
14. Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., and Anandkumar, A. (2018). Born Again Neural Networks. arXiv.
15. Zhang, Y., Xiang, T., Hospedales, T.M., and Lu, H. (2018, January 18–23). Deep Mutual Learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00454 ↗
16. Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., and Ma, K. (November, January 27). Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.
17. Yun, S., Park, J., Lee, K., and Shin, J. (2020, January 14–19). Regularizing Class-Wise Predictions via Self-Knowledge Distillation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01389 ↗
18. Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. (2020, January 26–30). AugMix: A Simple Method to Improve Robustness and Uncertainty under Data Shift. Proceedings of the International Conference on Learning Representations, Virtual Conference.
19. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 26). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
20. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2018, January 18–23). Learning transferable architectures for scalable image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00907 ↗
21. Real, E., Aggarwal, A., Huang, Y., and Le, Q.V. (2019, January 27). Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33014780 ↗
22. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q.V., and Wu, Y. (2019, January 8–14). Gpipe: Efficient training of giant neural networks using pipeline parallelism. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
23. Chorowski, J., and Jaitly, N. (2017, January 20). Towards Better Decoding and Language Model Integration in Sequence to Sequence Models. Proceedings of the Interspeech, Stockholm, Sweden.10.21437/Interspeech.2017-343 ↗
24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is All you Need. arXiv.
25. Xu, Y., Xu, Y., Qian, Q., Li, H., and Jin, R. (2020). Towards Understanding Label Smoothing. arXiv.
26. Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. (2017). Regularizing Neural Networks by Penalizing Confident Output Distributions. arXiv.
27. Yuan, L., Tay, F.E.H., Li, G., Wang, T., and Feng, J. (2019). Revisit Knowledge Distillation: A Teacher-free Framework. arXiv.
28. Self-Distillation as Instance-Specific Label Smoothing
29. Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Chi, E.H., and Jain, S. (2020). Understanding and Improving Knowledge Distillation. arxiv.
30. Krizhevsky, A., Nair, V., and Hinton, G. (2021, January 18). CIFAR-10 (Canadian Institute for Advanced Research). Available online: http://www.cs.toronto.edu/~kriz/cifar.html.
31. Darlow, L.N., Crowley, E.J., Antoniou, A., and Storkey, A.J. (2018). CINIC-10 is not ImageNet or CIFAR-10. arXiv.
32. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 11–14). Identity mappings in deep residual networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_38 ↗
33. Zagoruyko, S., and Komodakis, N. (2016). Wide Residual Networks, BMVC.10.5244/C.30.87 ↗
34. Musical genre classification of audio signals10.1109/TSA.2002.800560 ↗
35. Wang, L., and Yoon, K.J. (2021). Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks. IEEE Trans. Pattern Anal. Mach. Intell., 1.
36. Jin, X., Peng, B., Wu, Y., Liu, Y., Liu, J., Liang, D., Yan, J., and Hu, X. (November, January 27). Knowledge Distillation via Route Constrained Optimization. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.
37. Luan, Y., Zhao, H., Yang, Z., and Dai, Y. (2019). MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks. arXiv.
38. Song, G., and Chai, W. (2018, January 2–8). Collaborative learning for deep neural networks. Proceedings of the Advances in Neural Information Processing Systems, Montréal, QC, Canada.
39. Xu, L., Zhu, X., and Gong, S. (2018, January 2–8). Knowledge distillation by on-the-fly native ensemble. Proceedings of the Advances in Neural Information Processing Systems, Montréal, QC, Canada.
40. Müller, R., Kornblith, S., and Hinton, G.E. (2019). When Does Label Smoothing Help?, NeurIPS.
41. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report.
42. Improved music genre classification with convolutional neural networks